Subject: RE: [nfsv4] Directory delegations, take 2

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Oct 2003 07:38:24 -0700

> The purist in me would love to have directory delegations
> be symmetric with file delegations, but I grant that the
> complexity and recovery semantics of revoking a write
> delegation can get really ugly.

The purist in me would like that symmetry as well and we're 
not foreclosing it in some 4.x for x > 1, but I can't see
us dealing with the all the issues in any reasonable 4.1
time frame.

> I will claim that one of the set of requirements that we are
> trying to solve is the ability of a client to cache the
> contents of a directory (the names and file handles) in
> order to efficiently operate on part of the namespace tree.

> I will argue that the ability for a client to modify a
> directory without going to the server is not a requirement.
> For all the arguments laid out by Dave, it will be difficult
> to handle not only the error semantics, but it would imply that
> the client should have a reasonable ability to determine what
> the new cookies should be, otherwise we take the rare problem
> of invalid cookies and make it the common case.

> I would support read-only delegations of directory contents.
> All directory modifying operations MUST we written through to
> the server and any client having a delegation MUST be notified.
> In the unlucky event that a client is unreachable, the effect is
> that directory entries created or deleted by another client
> may go undetected.  But I will claim that we have this problem
> today in NFSv3 with normal client DNLC's and attribute caching,
> this just opens the window a bit wider in case of a lost revocation.

The other issue is that the client doing the modification must
wait for the unresponsive client, but since, as you point out,
the effects of giving up on the notification, and allowing the
update to proceed without it are not dire, we can be aggressive 
in allowing that to happen, and so the client will not have to
wait a very long time.

> I agree that the delegation recall (effectively change notification)
> should be synchronous. It should not be a large burden on the
> client as by definition there is nothing that will need to be
> written back, only a flush of cached entries.  It even leaves open the
> option for client to ignore the recall and continue to run out
> of their caches knowing that there may be an inconsistency. Seat belts?
> Don't need no stinking seat belts! To advocate async recalls will
> prohibit an interesting class of clients that need to know
> about changes.

If the client is going to ignore the recall, why he is he bothering
to get the delegation in the first place?  It seems a shame to order
those special seat-belts if you're not going to use them.  With a person,
it would probably be that he wasn't drunk when he ordered them, but 
does that apply to v4 clients?  Sometimes, at bakeathon, it seems like
my server has had few beers too many.  

> > Tom Talpey (private e-mail) has raised the issue of whether change notifica-
> > tion is worth it at all, and whether instead you just have a recall/revoca-
> > tion event and let the client just get his delegation again and refetch the 
> > modified directory contents.  This does make the feature easier to spec 
> > (e.g. There is a tough case of sequencing notifications and successive 
> > READDIR's when fetching a big directory, as well as a more complicated set 
> > of callbacks to define).  However, I worry about very large directories and 
> > the effort of refetching, when there is a modest level of exogenous
> > directory change.  What do other people think?  I think I'm going to go 
> > forward trying to do the notifications, unless it turns out  that the 
> > complexities make this too difficult to do for v4.1. 

> Assuming that as part of the addition of directory delegations, we
> add in the concept of reacquiring a delegation, I see no
> significant difference between a recall and a change notification.
> Both must occur on a directory modifying event, I can see an
> argument that a change notification can be made on a per file basis
> versus a delegation on a per directory basis. Because the order,
> cookies, and contents can have a ripple effect through the
> directory I am not sure that a client can safely do anything other
> than reread the directory. I am explicitly not addressing attribute
> changes here, more on that later.

The significant difference is that you don't have to re-read the
directory, in most (almost all?) cases, and when directories are
large (and many are) that is a big deal indeed.

You say that it might be the case that a single change could have a
ripple effect (order, cookies, etc.) and that is possible but not
very common.  Such a thing would cause clients doing multiple
READDIR's to get the whole directory to lose their mind.  The spec
tells servers to try not to do this, and I suspect most servers don't.

I agree that you do need the option of recall if you are in a situation
where the server for whatever reason has changed so much stuff that
re-READDIR is the best thing.  However, in most cases, something simple
has happened and the server knows that.  When a file is created and nothing
else changes in the directory, it makes sense for the server, which
knows that to tell the client, so he can get rid of is negative dnlc
entry, without rereading the whole directory, simply because some
other changes might have happened, when it in fact they didn't.

There is still room for argument about whether the benefits justify the 
complexity costs of notifications but there clearly is benefit, which
is pretty significant in many cases.

> > One issue that Saadia Khan raised is the issue of directory changes made 
> > by the delegate himself.  I think we have to make clear that this is 
> > allowed and the client is presumed to know about directory changes it 
> > makes itself.  Doing otherwise would compromise the usefulness of 
> > directory delegations in the case in which a single client is modifying 
> > the directory.  My assumption is that it just too difficult to do write 
> > directory delegation, but exclusive use is still a very important case, 
> > and we should what we can to make read directory delegations useful in 
> > the exclusive use environment.  

> I find this contradictory with your previous arguments against write
> delegations. All the issues of revocation and unresponsive clients
> exists in this scenerio as well.  The only way to make this sort of
> xclusive access work is to make irrevokable delegations which will
> suffer the problem of stale locks blocking any directory access.
> You must require all modifying directory operations to write through.

Yes, all modifying directory operations must be written through.  We
agree on that.  The question is whether the client who did that write
loses his delegation (or gets a notification).  I'm saying he doesn't.
He is presumed to know about changes he makes himself (by getting
the reply) and he retains the delegation and thus the ability to be
notified (whether through change notifications or recall) about 
changes made by other clients.

> On the argument against a separate attribute delegation it appears
> to come down primarily to that of performance. One recent a-ha that
> I had with respect to the old READDIRPLUS as translated to v4,
> the straight forward implementation of requesting the desired
> attributes as part of the READDIR operation may not be the most
> efficient mechanism to process from a client perspective. With
> the availablity of COMPOUND, the client can alternatively issue
> a READDIR with no requested attributes (trivial decode) and then
> issue a single COMPOUND of LOOKUP's for all the names returned.
> (Alternatively request just the nfs_fh4 and COMPOUND GETATTRs).

Clearly you could do this and the performance might not be terrible
but you seem to claim ("the READDIR operation may not be the most
efficient mechanism to process from a client perspective") that
this can perform better or at least equal to getting the attributes
with READDIR and I don't see how that possibly could be true.  
Besides the latency issue of two round-trips vs. one, the client is
going to have to process the directory and turn it into a large
compound and that is going to take time.  The client would also
have to deal with the fact that the directory could change before
the second compound, and one of the LOOKUP's/GETATTR's could fail,
breaking the chain, and requiring another COMPOUND be issued.

If there is going to be a proposal to get rid of READDIR in v4.1 
(or more accurately make it mandatory to not implement) and add a 
simplified version (READDIRMINUS?), I'm definitely against it.

> With this sort of approach, if the client wanted to gather
> delegations of attributes it could just COMPOUND the delegation
> requests with the GETATTRS, likewise revocation could also be
> COMPOUND'd. From a performance perspective this seems trivially
> more work than doing it on a per directory basis.

I think this depends on your definition of "trivially".  Clearly 
this is possible, but it can get expensive.  Big compounds have a
lot of interpretation overhead.  Doing a thousand ops is not as
bad as doing a thousand requests but it takes more time than doing
a single op.

For me the big issue is that client and server have to keep track
of a lot of separate stateid's for each of these files.  In 
environments where you have tons of files that change rarely, 
I'm worried about the overhead of adding all that state management.
It seems better to me to aggregate it so that it is more manageable.

> More importantly, unless we want to ban hard links, or delegation of
> directories that contain files with hard links (something I would
> probably support notwithstanding Posix), determing which directory
> delegations need to get revoked due to a SETATTR on a hard linked
> file is incredibly hard to implement.

First point, even it were the case, that determining which delegations 
to recall was very hard in the presence of hard links, you would
not have to ban such delegations.  Delegations, as I see it, are
optional for the server, and if that server would have trouble, 
providing the requisite guarantees for a particular directory, such
as one with files that had hard links, then it is perfectly free
to return some sort of NFS4ERR_SORRY_NO_CAN_DO.

As to the difficulty of this, let me turn your COMPOUND argument
back on you.  You are say that if I send back information on each of
the files and that the client asks for a delegation for each,
that is OK, i.e. not so hard to implement.  On the other hand, if
the server in response to a directory delegation does exactly the
same operations it would have done had the client requested all
those individual delegations (the only difference is that the
client doesn't have to do individual delegation requests for files
and that he only has a single delegation stateid instead of one
for each file), the task of notifying the clients on a SETATTR of
a hard-linked file has become difficult.  Why?  I can't see
anything that would make this harder for the server.  On the other 
hand, in many cases, such as where there are no or few hard links 
in the directory, it might be considerably easier.  The point is that
with the semantic of attributes being guranteed by the directory 
delegation, the server can take adavantage of those common, easier
cases, to do the simpler thing, if that is available to him. 

> I have not thought of how to spec this in a protocol, but Mike's
> idea's of OPENDIR/CLOSEDIR looks interesting...

> While I can see how we could nicely add in symlink delegation,
> I am not sure that is a "problem" that needs to be solved.

Since Carl is the one with a need for this, I'll leave this to him.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 22 Oct 2003 09:35:13 -0700

> us dealing with the all the issues in any reasonable 4.1

The "everything is a file" purists long ago gave up on directories as
files.  Directories may look like files on disk (or not), but they're
conceptually not just like files.

Cheers,

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 22 Oct 2003 09:58:24 -0700

[...]
> in allowing that to happen, and so the client will not have to

Synchronous notification delegations sound both neat and like trouble.
If we can't guarantee synchronicity (and from these arguments it's clear
that we shouldn't), what good is it to provide synchronous notification
delegations?  Applications will not be able to use the notification
mechanism for synchronization without the guarantee, so why would they
want synchronous notifications?

[...]
> other changes might have happened, when it in fact they didn't.

This is clearly a rehash of the same discussion that has surely taken
place in the design of many a database replication protocol: "we want to
be efficient for the common case of low flux rates, but we don't want
periods of large flux rates to cause replication bandwidth to zoom past
the bandwidth necessary to just send the database; during periods of
medium flux rates we want to batch up notifications close to each other
in time, and during periods of large flux rates we want to throttle down
replication and replicate the whole DB instead of incremental updates."

We're really talking about replication of lots of mostly small
databases, where the directories are the databases.

And note that where a client is interested in getting notifications for
many directories in a file system, not just one, a per-directory
notification system will be inefficient, so a way to get a notification
delegation for all directories in a file system may be a good idea.

Note that if we take this to extremes we get an [async] file system
replication protocol :)

> changes made by other clients.

Whether it gets a notification or not is not as interesting as: making
sure that the server can know not to send a notification, if the client
must not get one, or making sure that the client can tell that it was
the source of some notification so that it can ignore it.

Cheers,

Nico
--

---

From: Ted Anderson <[email-masked]>
Date: Thu, 23 Oct 2003 09:20:32 -0400

On 10/21/2003 16:41, David Robinson wrote:
 > I will claim that one of the set of requirements that we are trying to
 > solve is the ability of a client to cache the contents of a directory
 > (the names and file handles) in order to efficiently operate on part
 > of the namespace tree.

Yes, and not just efficiently, but with well-defined semantics.

 > I will argue that the ability for a client to modify a directory
 > without going to the server is not a requirement.  For all the
 > arguments laid out by Dave, it will be difficult to handle not only
 > the error semantics, but it would imply that the client should have a
 > reasonable ability to determine what the new cookies should be,
 > otherwise we take the rare problem of invalid cookies and make it the
 > common case.

I even agree with this.  I think in the long run, providing very high
scalability and client performance will require write delegation of
directories, but there are significant complications.  So putting this
off makes sense.

 > I would support read-only delegations of directory contents.  All
 > directory modifying operations MUST we written through to the server
 > and any client having a delegation MUST be notified.  [...]  I agree
 > that the delegation recall (effectively change notification) should be
 > synchronous. ...

Yes.  I think this would be sufficient to provide very useful semantics
to clients.  Anything less than synchronous notifications leads to
update semantics that whose specification is too fuzzy to be very
useful.  Asynchronous notification avoids polling and the performance
problems associated with that approach.  But when you consider delivery
delays, there is essentially no improvement over the traditional
time-based approach WRT consistency semantics.  Making the notifications
synchronous provides are qualitative improvement in the protocol.

On 10/22/2003 12:58, Nicolas Williams wrote:
 > If we can't guarantee synchronicity (and from these arguments it's
 > clear that we shouldn't), what good is it to provide synchronous
 > notification delegations?  Applications will not be able to use the
 > notification mechanism for synchronization without the guarantee, so
 > why would they want synchronous notifications?

Nico wades in on the other side, due to concern about failures.  My
counter argument is that failures can be handled effectively while
preserving the benefits synchronicity provides.

It is certainly possible to make the notification timeout smaller for
read delegations than for write delegations.  Bounding the time required
to send back cached updates is hard for write delegations, but easy for
read delegations, which require relatively small amounts of work on the
client.  The important aspect of the timeout, however small it is made,
is to ensure that it is tied to the lease interval.  In this way the
client always knows when the server might have failed to deliver a
delegation related message.  This has the big advantage that consistency
semantics are preserved as long as there are no failures of this sort.
These failure events can be delivered to highly paranoid applications or
can be considered after the fact when debugging application problems.

There are still interesting issues to debate.  I like the idea of
delivering notifications which include a description of the directory
update so that delegations do not have to be recalled.  It seems to have
large performance benefits in the common case of low-flux updates.
However, Nico's discussion of "directories as databases" is germane.

The topic of granularity is also very interesting and important.  Should
a directory delegation apply to its contents only or also the attributes
of its children?  Since a file delegation can protect file attributes,
extending directory delegation to its children is not strictly necessary
from a functional point of view.  However, the benefits of reducing the
state management burden are not to be overlooked.  Once you consider
children, it is tempting to allow entire subtrees as well (again,
hardlinks rear their ugly heads here).  A delegation on an entire file
system would also be a very nice feature for when the update-flux is low
enough, as it often is.

I am mindful of the complexity and timing concerns of addressing all
these additional issues.  At a minimum, I think read directory
delegation with synchronous recall is necessary to provide a useful
increment in functionality over NFSv4.0.

Ted Anderson

---

From: David Robinson <[email-masked]>
Date: Mon, 27 Oct 2003 22:13:29 -0600

[Catching up on last week's thread...]

Directory delegations in this round should be read-only.

Why allow no seatbelts? My point was that with a delegation
a client knows the state of its consistency, when asking
for a delegation but ignoring the callback it knows when
its cache is consistent and when it won't be. Without a delegation
it has no way of knowing. Bottom line, this becomes a client
quality of implementation issue, not a protocol issue so we can
ignore it as a rant on my part.

In recall versus change notification, with a read-only delegation
a recall operation in the sense of a delegation is just a notification
that the directory has changed as there is no data that will
ever need to be written back, at worst it is just a cache flush
notification. Anothedr way of putting it is that what we have
been calling a "recall" is just a change notification on all
entries. As one approach, we could design this so that we
allow a change notification operation that takes a directory
and a list of cookies, one for each entry that changed.  Then
have the special zero cookie indicate all entries changed as
an optimized method.

Sync vs async, the issue is not timeliness, in both cases it is
hard to guarentee timeliness.  But as Dave pointed out, the
primary advantage of sync is in allowing ordering of operations.
We MUST have sync notifications, but it may be useful to have
on acquistion of a delegation the client specify if it wants
sync or async notifications.  Some clients may not be able to do
anything better with a sync notification so they may not want to
impose a burden, while others may need it for interesting applications.

Directory operations must be written through, there is no point in
requiring the client to re-read what it just wrote or for a
server to callback to the client that sent it the change, the client
knows the current state.

I understand the performance impact of not including the attributes
into the delegation.  My major concern is in the area of scaleability.
Currently in most Unix systems, data and metadata operations on a file
require no locks or interactions with any other file, it is highly
scaleable. Likewise naming operations only involve the directory
involved and do not require locks on component files.  Tying the two
could involve coordinating a large number of locks which will
negatively effect scalability.  If we keep attribute delegation
separate and allow per entry change notification, the impact of
a directory changing will be modest and the change of a file's
attributes would not ripple through other files.

	-David