Subject: Spec updates

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 2 Aug 2002 10:24:02 -0700

Here is the input to deal with the important delegation and migration 
issues:

     50, 83, 96, 97, 98, 104, 105, 112, 113

So this leaves on my high-priority list:

     51, 57, 61, 72, 81, 92, 93, 101, 114, 129, 142, 178, 
     181, 186, 187, 191, 193, 194

minus whatever you have done.  If you were willing to part with
information about what you have done so far, I could pick up some
of these.  Let me know. 

------------------------------------------------------------

Issue 50: Clarify the behavior of a RENAME of a file that is open and has 
an outstanding delegation. Need to describe the problems fully and describe 
potential solutions - server recalls delegation at the point a RENAME is 
attempted. The server can choose to add this type of delegation recall when 
a parent directory is renamed along with support for the target file itself. 

     I think this is reasonable to do.

>>> Discussion:

I just looked at this.  There is a bullet item saying that rename of a 
file as the source or target necessitates delegation recall.  Then the
following paragraph describes the issue with rename of the directory
and says that if server semantics would deny the rename if the file is
opened, it must recall the delegation to find out if the file is in fact open.

>>> Resolution:

** No change.
 
-----------------------------------------------------------------

Issue 83: Clarify what happens when OPEN returns a delegation and the client
does not do a OPEN_CONFIRM. For example, conflicting OPEN is received, does 
the server recall the delegation? Or client returns the delegation before 
OPEN_CONFIRM, what error is returned, if any. What happens if the client 
specified the delegation in an OPEN with claim type DLG_CUR? Does this 
confirm the delegation, or is it invalid? 

     I need to address this. 

>>> Discussion:

This can get quite nightmarish.  I think discretion is the better part of
valor here.  So my resolution is that a server should simply not grant 
delegations when the associated open is subject to a confirmation requirement.
There's a couple of issues that need some informal discussion before I
proceed to the spec changes.

First note that the spec doesn't currently say anything about the issue of
conflicting opens and unconfirmed (or not-yet-confirmed opens).  Suppose
the server gets a DENY=WRITE open which requires confirmation, and while
it is a awaiting that confirmation, an open for write comes in.  I don't 
think the spec as written requires you to wait for the confirmation issue
to be resolved, although it might be a nice thing to do, because otherwise,
the guy doing the write might get a share-denied when he shouldn't.

While this might be a nit for regular open, it would be horrendous for
the reclaim case, because getting a spurious deny would be catastrophic 
when you already have the file open. While it is possible to wait for the 
lease period to expire to resolve, a simpler solution would simply be not 
to return unconfirmed opens during the grace period at all.  The server 
should not have the time to drop lockowners when it is just starting up.
Where this ties into the the current problem is that reclaim is one situation
in which you can ask for a delegation and the server has to do it.  So if,
that were allowed to requir open-confirm then we couldn't exclude delegations
coming along with unconfirmed opens.  However, I think we can and should say
that confirmation-required opens cannot happen on reclaim and will never 
happen when a delegation is coming along.

>>> Resolution:

** Take the last paragraph from section 8.1.7 and make it the first
   paragraph of a new section 8.1.8.

8.1.8 Use of Open Confirmation

** Insert last paragraph of 8.1.7 here and then proceed as follows:

There are a number of situations in which the requirment to confirm
an OPEN would pose difficulties for the client and server, in that 
they would be  prevented from acting in a timely fashion on information 
received, because that information would be provisional, subject to 
deletion upon non-confirmation.  Fortunately, these are situations
in which the server can avoid the need for confirmation when responding
to open requests.  The two constraints are:

o    The server must not bestow a delegation for any open which would 
     require confirmation.

o    The server must not require confirmation on reclaim-type open
     (i.e one specifying claim types CLAIM_PREVIOUS or CLAIM)DELEGATE_PREV).

These constraints are related in that reclaim-type opens are the only
ones in which the server may be required to send a delegation.  For 
CLAIM_NULL, sending the delegation is optional while for CLAIM_DELEGATE_CUR,
no delegation is sent.

Delegations being sent with an open requiring confirmation are troublesome
because recovering from non-confirmation adds undue compelxity to the 
protocol while requiring confirmation on reclaim-type opens poses
difficulties in that the inability to resolve the status of the reclaim
until lease expiration may make it difficult to have timely determination 
of the set of locks being reclaimed (since the grace period may expire).

Requiring open confirmation on reclaim-type opens is avoidable because of
the nature of the environments in which such opens are done.  For 
CLAIM_PREVIOUS opens, this is immediately after server reboot, so there
should be no time for lockowners to be created, found to be used, and
recycled.  For CLAIM_DELEGATE_PREV opens, we are dealing with a client
reboot situation.  A server which supports delegation can be sure that
no lockowners have been recycled since client initialization and thus
can ensure that confirmation will not be required.

** Rewrite the third paragraph of the IMPLEMENTATION section of 14.2.18
   as follows:

Servers must not require confirmation on OPEN's that grant delegations
of are doing reclaim operations (See section 8.1.8 - Open Confirmation 
for details).  The server can easily avoid this by noting whether it
has disposed of one nfs_lockowner for the given clientid.  If the server
does not support delegation, it might simply maintain a single bit that 
notes whether any nfs_lockowner has been disposed of.

----------------------------------------------------------------------------

Issue 96: Section 9.2.1 uses CLAIM_DELEGATE_PREV when CLAIM_DELEGATE_CUR 
should be used when referring to recovery of delegations after server failure. 

     Don't agree with this statement, but something needs to be fixed
     here.  There is confusion about CLAIM_DELEGATE_PREV.  I'll come up
     with a proposal.

>>> Discussion:

Recovery of delegations after server failure should use CLAIM_PREVIOUS.

>>> Resolution:

** In the first sentence of the penultimate paragraph (excluding 
   bulleted paragraphs), change the last word in the parenthesized
   participial phrase from CLAIM_DELEGATE_PREV to CLAIM_PREVIOUS. 

-------------------------------------------------------------------------

Issue 97: Section 9.4.2 needs an addition that speaks of read delegations 
and the fact that the client should send read lock request to the server 
for arbitration. 

     We can add something that says this.

>>> Discussion:

In fact, you don't have to do this.  If exclusive locks require you to
have the file open for write, then the existence of a read delegation,
means that nobody can have the file open for wirte which implies nobody
could issue an exclusive lock, so we could do locks (shared only) with a
read delegation, but the value of this is small.  The spec complexity 
just isn't worth it.

>>> Resolution:

** Add a new paragraph at the end of section 8.4.2 which says:

When a client holds a read open delegation lock operations are not
performed locally.  All lock operations, including those requesting
non-exclusive locks, are sent to the server for resolution. 

------------------------------------------------------------------

Issue 98: In section 9.5.1 there is a discussion of what a client should 
do if its delegation is revoked. It says that unless the client can 
guarantee that nobody else has modified the file, it must not write back 
a copy of the local bits unless it has the entire file cached. Rationale 
for this needs to be provided to clarify the scenario that is being protected 
against. Should the RFC forbid the client from saving a partial file?

     This can be slightly relaxed I suppose.

>>> Discussion:

** In the last paragraph, replace everything past the second sentence 
   with the following:

Note that when the client can determine that the file not been modified 
by any other client, or when the client has a complete cached copy of
file in question, such a saved copy of the client's view of the file
may be of particular value for recovery.  In other case, recovery using 
a copy of the file based partially on the client's cached data and partially
on the server copy as modified by other clients, will be anything but 
starightforward, so clients may avoid saving file conetnts in these
situations or mark the results specially to warn users of possible
problems.

Saving of such modified data in delegation revocation situations
may be limited to files of a certain size or may only be used when 
sufficient disk space is available within the target file system.
Such saving may also be restricted to situations when the client has
sufficient buffering resources to keep the cached copy available
until it is properly stored to the target file system. 

--------------------------------------------------------

Issue 104: Review section 9.4 to ensure that the use of recall and revoke 
are used in the appropriate context and consistently as defined for the 
NFSv4 usage. 

     I think this is OK but I can address this when I'm in this area. 

>>> Discussion:

I scanned for every use of these words in the entire spec.  They are OK.

>>> Resolution:

** No change.

----------------------------------------------------------------------

Issue 105: Clarify the use of CLOSE (with delegation stateid) as a means 
for the client to return a delegation to the server. Is the client required 
to CLOSE for both the original OPEN and the delegation stateid or is the 
delegation stateid CLOSEd when the CLOSE is done for the OPEN stateid? Or 
should the client CLOSE with the stateid returned from OPEN as well as 
DELEGRETURN with the delegation stateid? 

     The last of these is correct.  I don't see the problem.  Delegations
     are returned with DELEGRETURN and OPEN's are bracketed with CLOSE's.
     What more needs to be said?

>>> Discussion:

As Mike and I discussed, it just needs to be made clearer that the 
fundamental lock is the delegation and that the only constraint is that
the issue of wether the file is really open has to be resolved on the
server *before* the delegation is returned.

>>> Resolution:

** Add a new paragraph at the end of section 9.4.3, which says:

An implementation need not wait until delegation recall (or deciding
to voluntaily return a delegation) to perform any of the above actions,
if implementation considerations (e.g. resource availability constraints)
make that desirable.  Generally, however, the fact that the actual
open state of the file may continue to change makes it not worthwhile
to send information about opens and closes to the server, except as
part of delegation return.  Only in the case of closing the open that
resulted in obtaining the delegation would cleints be likely to to
this early, since, in that case, the close once done will not be
undone.  Regardless of the client's choices on scheduling these
actions, all must be performed before the delegation is returned,
either in previous reequests in or is previous operations in the
same COMPOUND request.

--------------------------------------------------------------------------

Issue 112: For filesystem migration, need to add a discussion of what 
will happen to the lease period at the new server and what the client 
SHOULD/MUST do to obtain that lease period. If the server migrated the 
filesystem and the related state, does the new server have to maintain 
both its original lease period and the lease period for the filesystem 
that was just moved under its control? If not, then is the client 
responsible for obtaining and respecting the new lease period after it 
contacts the new server? The new server must wait for at least the lease 
period of the old server before making a switch to a new duration to 
allow clients migration to the new server be achieved. 

     I would say there are two cases here.  If the migration operation 
     transfers state, then I think the lease time should be the same
     and the client doesn't have to do anything.  If he accesses the
     new server and the state isn't transferred, then he had to do
     that for a reboot.  (Issue 144 deals with this).  I think we can 
     write a few sentences that makes this clearer.

>>> Discussion:

Nothing to add to the above.

>>> Resolution:

** Add a new section after section 8.13.3, as follows:

8.13.4. Migration and the Lease_time Attribute

In order that the client may appropriately manage its leases in the
case of migration, the estiantion server must establish proper values 
for the lease_time attribute.

When state is transferred transparently, that state should include
the correct value of the lease_time attribute.  The lease_time
attribute on the destination server must never be less than that 
on the source sicne this wuld result in premature expiration of
leases granted by the source server.  Upon migration in which state 
is transferred transparently, the client is under no obligation to
re-fetch the lease_time attribute and may continue to use the value
previously fetched (on the source server).

If state has not been transferred transparently (i.e. the client sees
a real or simulated server reboot), the client should fetch the
value of lease_time on the new (i.e. destination) server, and use it
for subsequent locking requests.  However the server must respect a 
grace period at least as long as the lease_time on the source server,
in order to ensure that clients have ample time to reclaim their
locks before potentially conflicting non-reclaimed locks are granted.
The means by which the new server obtains the value of lease_time
on the old server is left to the server implementations.  It is not
specified by the NFS-v4 protocol.

-----------------------------------------------------------------------

Issue 113: CB_GETATTR of "change" attribute needs to be clearly defined 
with respect to the client's update of this "opaque" attribute.

     I think the principle needs to be that the delegated client will
     minimally update change number (increment it once per cb_getattr
     call that encounters a change made since a previous cb_getattr
     call) and that the server will take account of these updates
     in re-establishing its own control of this attribute.  Now we just
     figure out to say that (didn't I just do what?  Well not exactly).

>>> Discussion:

Nothing to add to the above.

>>> Resolution

** Add the following paragraph after the fourth (unbulleted) paragraph
   of section 9.4.3 (The one beginning "The server needs"):

Since the form of the change attribute is determined by the server and
is opaque to the client, the client needs to take special care in
generating change attribute values in response to a CB_GETATTR callback.
Since clients assume a monotonically increasing change attribute value,
the delegated client should not increment the change attribute to a greater
degree than absolutely necessary, allowing the server after delegation
return to resume its control of the change attribute, without interupting
its monotonic increase.  The delegated can provide this minimally 
disruptive handling of the change attribute by only incrementing the
change attribute value by one when it recieves a CB_GETATTR callback 
that encounters a change in the file since the previous CB_GETATTR
callback (or receiving of the delegation).  The server should then be 
able to take acount of these increments that it noted in the responses
to the CB_GETATTR's and resume its own control of the change attribute.