Subject: Getting to Proposed Std, again (Chapter 2)

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 16 Jul 2002 12:36:50 -0700

Issue 94: Clarify the responsibility of the client to "clean up" state at
the server for operations that may be aborted at the client. For example, 
if an application is awaiting a lock request and that application is killed 
at the client (Unix ^C), the client is responsible for ensuring that lock 
request in progress with the server is accounted for. This means that the 
client may release the application so that it may be stopped before receiving 
a response from the server for the lock request but an appropriate UNLOCK 
and CLOSE must be done. There are other places where this type of state 
clean up is required of the client.  

     I think going in this direction is a mistake.  It gets way to much
     into the details of client implementations.  The principle is that
     if you do something that creates state you have to undo that.  The
     whole thing about applications and control-C is solely within the 
     purview of the client implementation.

Issue 95: Additional clarification for section 8.13.3 that discusses under 
what conditions NFS4ERR_LEASE_MOVED and NFS4ERR_MOVED are returned. Needs to
be stated up front that NFS4ERR_LEASE_MOVED is used to signify a migration of 
"some" filesystem at the server. 

     I don't see what's wrong with what is there.

Issue 96: Section 9.2.1 uses CLAIM_DELEGATE_PREV when CLAIM_DELEGATE_CUR 
should be used when referring to recovery of delegations after server failure. 

     Don't agree with this statement, but something needs to be fixed
     here.  There is confusion about CLAIM_DELEGATE_PREV.  I'll come up
     with a proposal.

Issue 97: Section 9.4.2 needs an addition that speaks of read delegations 
and the fact that the client should send read lock request to the server 
for arbitration. 

     We can add something that says this.

Issue 98: In section 9.5.1 there is a discussion of what a client should 
do if its delegation is revoked. It says that unless the client can 
guarantee that nobody else has modified the file, it must not write back 
a copy of the local bits unless it has the entire file cached. Rationale 
for this needs to be provided to clarify the scenario that is being protected 
against. Should the RFC forbid the client from saving a partial file?

     This can be slightly relaxed I suppose.

Issue 99: Section 9.7 says not to do negative LOOKUP caching. But if we're 
doing READDIR caching, shouldn't it be okay to have a negative LOOKUP cache 
entry, as long as it expires when the READDIR cache expires? 

     I think the intention was for OPEN's.  In the case of positive
     name caching, it is clear you can't use this for OPEN's, but for
     non-existence you might be tempted to avoid the OPEN attempt and
     I don't think that would be right.  Anyone else have opinions 
     about this?

Issue 100: Add to description of SECINFO usage (integrity) and initiation 
of secure channel in section 3. 

     Not sure what is being requested here.

Issue 101: Add section describing stateid:seqid:lockowners and state 
recovery to RFC3010. 

     If we are going to do this, we need a volunteer.

Issue 103: In section 14.2.11 under IMPLEMENTATION, the wording about 
LOCKT using a lockowner and LOCK using stateid is awkward. Should be changed. 

     OK.  How about:

     LOCKT uses a nfs_lockowner4 rather a stateid4, as is used in
     LOCK to identify the owner.  This is because the client does not 
     have to open the file to test for the existence of a lock, so
     a stateid may not be available.

Issue 104: Review section 9.4 to ensure that the use of recall and revoke 
are used in the appropriate context and consistently as defined for the 
NFSv4 usage. 

     I think this is OK but I can address this when I'm in this area. 

Issue 105: Clarify the use of CLOSE (with delegation stateid) as a means 
for the client to return a delegation to the server. Is the client required 
to CLOSE for both the original OPEN and the delegation stateid or is the 
delegation stateid CLOSEd when the CLOSE is done for the OPEN stateid? Or 
should the client CLOSE with the stateid returned from OPEN as well as 
DELEGRETURN with the delegation stateid? 

     The last of these is correct.  I don't see the problem.  Delegations
     are returned with DELEGRETURN and OPEN's are bracketed with CLOSE's.
     What more needs to be said?

Issue 106: Clarify the mandatory locking flag returned on OPEN and how it 
is to be used (pp 77 of rfc3010 discusses some of it). Is it an indication 
of existing locks for the file or what services the server is capable of 
based on its implementation or server filesystem? 

     It's an indication of the type of locking that will be implemented.
     The wording "that mandatory locking is in effect" in the DESCRIPTION 
     for OPEN is misleading.  Let's substitute "whether mandatory (as
     opposed to advisory) locking is implemented by the server".

Issue 107: Large clarification/addition needs to be made for the use of 
stateids and the NFSv4 state model in general.

     I'm sure things could be made clearer in places but we're not 
     rewriting it. 

Issue 108: Clarify minor versioning mechanism (add operations only and use 
by client of various versions); also, operations in previous minor versions 
are implemented as well, just not the new ones. 

     I don't understand what's being asked for.  The minor versioning
     section has lot of detail.  I don't see the problem.

Issue 109: Recovery of 'out-of-sync' errors is not defined in the protocol. 
Should there be a method to remove all state associated with a lockowner 
based on an implementation error (either in client or server)? 

     There's RELEASE_LOCKOWNER and it would allow this I suppose.  It
     doesn't have any sequence so could act as a big hammer.

Issue 110: Error ordering for determining if a lease has been implicitly 
renewed as part of a request that receives an error. Need to work through 
the lease renewal logic and propose a methodology for RFC updates.

     Let's not go down this road.  It would be extremely burdensome on
     the spec (authors) and on server implementations.  All the client is
     entitled to assume is that if the operation which would have done 
     the renew is after the operation which got the error it didn't 
     happen.  If it was before the operation which failed, it was
     renewed.  If it was the operation which failed all bets are off.
     Maybe, maybe not.  The client can do a simple RENEW if he wants 
     to avoid errors.  Also note, that having the renew happen when
     you weren't expecting it should pose no problems for the client.
     All of which is about what you can determine from the current
     spec so let's leave well enough alone.

Issue 111: Implementation recommendation, at the least, should be made 
for how a client chooses its clientids. The client should use a different 
clientid for each server interface/IP address so that the client and 
server are not confused for delegation identification (client has same 
portion of the server namespace mounted in two locations at the client 
and from two different server interfaces). If the client sends the same 
clientid, the server will think that it is the same client (when dealing 
with delegations) and the client will not see the appropriate behavior 
with respect to delegation recall.

     The underlying issue here is the same as issue 114, so I'm consider 
     this a dup, and discuss things under that issue.

Issue 112: For filesystem migration, need to add a discussion of what 
will happen to the lease period at the new server and what the client 
SHOULD/MUST do to obtain that lease period. If the server migrated the 
filesystem and the related state, does the new server have to maintain 
both its original lease period and the lease period for the filesystem 
that was just moved under its control? If not, then is the client 
responsible for obtaining and respecting the new lease period after it 
contacts the new server? The new server must wait for at least the lease 
period of the old server before making a switch to a new duration to 
allow clients migration to the new server be achieved. 

     I would say there are two cases here.  If the migration operation 
     transfers state, then I think the lease time should be the same
     and the client doesn't have to do anything.  If he accesses the
     new server and the state isn't transferred, then he had to do
     that for a reboot.  (Issue 144 deals with this).  I think we can 
     write a few sentences that makes this clearer.

Issue 113: CB_GETATTR of "change" attribute needs to be clearly defined 
with respect to the client's update of this "opaque" attribute.

     I think the principle needs to be that the delegated client will
     minimally update change number (increment it once per cb_getattr
     call that encounters a change made since a previous cb_getattr
     call) and that the server will take account of these updates
     in re-establishing its own control of this attribute.  Now we just
     figure out to say that (didn't I just do what?  Well not exactly).

Issue 114: For the delegation section, the following discussion and 
implementation suggestion needs to be added. [long paragraph relating
to delegation confusion in the case in which the same fs is mounted 
twice by the client].
 
     I think we have a better solution to this.  The fact that we now
     have a special id (for callback identification purposes) that says
     which of our (the client's) servers this is allows this to be dealt 
     with better.  What has to be written are the rules for SETCLIENTID
     (e.g. new setclientid with same client name/verifier and different
     opaque id results in new clientid but now client reset, same client
     name with different verifier reset all clientid's with that name,
     irrespective of opaque id, same client name verifier/opaque id
     does nothing, and same client id/verifier with different callback
     information, exclusive of the opaque id is an error.  (This would
     also take care of issue 86 and 175). 

Issue 115: Clarify what should happen when a client has a read delegation 
from a server and then has a local application open the file for write. 
Should the client return the delegation and then send the OPEN(write)? 
Should the client attempt the OPEN(write) and let the server decide to 
return a write delegation or recall the delegation as a result of sharing? 
If the server is allowed to "upgrade" the delegation, how is that communicated
and the various delegation stateids resolved later on? 

     The real issue hear is whether there is an upgrade path for a 
     read delegation to a write.  I hadn't made provision for one, but
     it would seem easy to provide.  If a sever is returning a write
     delegation and the stateid is the same as the existing stateid for
     a write delegation, then the client can consider it an upgrade.
     What do people thing about adding that now (it being kind of late 
     in the day)?  As to the rest, I say do what you want.  Returning 
     the delegation seems like a good idea, waiting the recall would
     also work.  If there is an upgrade path, doing the delegation 
     return isn't critical but you can still do it.  

Issue 117: Clarify that delegations may be returned at will by the client 
(do not have to wait for the server to recall). 

     Add the following paragraph to the DESCRIPTION:

     Delegations may be returned when recalled or voluntarily (i.e. before
     the server has recalled them).  In either case the client must 
     properly propagate state changed under the aegis of the delegation
     to the sever before returning the delegation.

Issue 118: Need to clarify what type of objects are acceptable for the 
LINK operation. Directory, file, named attributes?

     I'd just leave this alone and put it under fs semantics, at least 
     given the schedule.

Issue 119: VERIFY/VERIFY implementation sections need to mention what to 
do with attributes like rdattr_error and time_access_set/time_modify_set. 
Should an error be generated, these attributes ignored? 

     I'm treating this as a DUP of Issue 53.

Issue 120: Clarify if SECINFO's list is ordered and should it be 
interpreted by the client as the server's preference of how the client 
should use the security mechanisms; if so, the client would then overlay 
its own policy.
  
     The spec doesn't indicate that.  Why would you asssume it?  As
     far as the issue of whether this is this case, I think its a
     bad idea.  If the server has a list of acceptable mechanisms
     and the client can choose among them, that's well-defined.  If
     the server has preferences and the client has preferences then
     you get into defining how these interact.  Is there a real 
     need for this?

Issue 121:

     The issue is that there is no issue.  Works for me. 

Issue 125: Client's REMOVE of an open file still requires the use of 
a RENAME to a .nfsXXXX file followed by a subsequent REMOVE when the 
client's applications release final reference to the open file. 

     Yup.  That stinks, but it's too late now.

Issue 129: Add discussion/definition text to differentiate between 
POSIX-style locking and Strict-locking. 

     This is worth a paragraph.

Issue 131: Clarify when the lease period starts at the server (SETCLIENTID). 

     The definition of Lease says it is the period you are irrevocably
     granted a lock, so it starts when you have one (including an open),
     and can be renewed.  So it starts when you don't have a lease and 
     do an OPEN and get a stateid.  Let's leave the spec as it is.

Issue 132: Clarify a few points about named attributes.
 
Named attribute discussion should include the possible implementation 
around the SETATTR of the attribute directory and named attributes. 

Because of potential overhead, the client should avoid overuse 
GETATTR("named_attr"). 

     The first is a dup of Issue 136.

     The second doesn't seem like something we should do.  What is "overuse"
     and why should I worry about potential overhead as opposed to actual
     overhead.  Let's not clutter up the spec with this.

Issue 133: Is exclusive create for named attributes reasonable and is it 
likely to be implemented? 

     I'd prefer to avoid it if we can and just define this as invalid.
     What do other people think?

Issue 136: Clarify operations (SETATTR/GETATTR) appropriate for named 
attribute directory (as returned by OPENATTR) and the effects of SETATTR 
on named attribute files (limited attribute support possible).

     I'm going to advance a hard line here and see if the group wants to
     push back.  The attribute directory is a way of presenting the set of
     named attributes and from the viewpoint of the protocol doesn't
     *really* exist.  So doing a SETATTR on it is an error.  As far as 
     GETATTR, attributes may be limited by the fact that there is no
     way to set them.  What does the group think?

Issue 138: Is the filehandle type changeable by the server? Is it possible 
for the server to switch from persistent to volatile filehandles on an 
administrative action? For example, can the re-exporting of a directory 
change the filehandle type based on export options provided by an 
implementation. If so, client recover from such events may be very 
difficult. 

     My translation of "very difficult": @#$%$# impossible.  If the
     server exports handles as persistent it is promising to honor 
     them forever (or until they become STALE): they can't expire.
     The server can't say: "Now I'm promising nothing and my previous
     promises don't count".  It can but it shouldn't do that.  I don't
     see this ever happening.  The form of handles for persistent and
     volatile handles, if a server supports both, are going to be 
     different, so that what would happen is that the old handles
     (persistent type) will get NFS4ERR_STALE while other clients
     will get new-form volatile handles.  No spec change for this 
     is needed.

Issue 139: When a client recovers and the "change" attribute has changed 
for a file and the client holds modified pages for that file are the 
following recovery actions acceptable for a client implementation: 

 o if the client had a write delegation for the file, mark the file as 
   unusable 
 o if the client did not have a write delegation and did not have any locks, 
   (try to) write back the dirty pages and flush the cache 
 o if the client didn't have a write delegation but did have locks, mark 
   the file as unusable 

     These seem reasonable except how is the client going to mark the
     the file unusable?

Issue 140: In the case of client recovery, are there any other attributes 
that the client should inspect when reopening a file 

     Not that I know of.  I think that's the point of the change attribute.

Issue 141: Are there issues with respect to directory entry ordering 
requirements for replication and migration to work properly. For example, 
if the directory ordering changes on a migration event, the client may 
not be able to continue from READDIR operations that have not fully read 
a directory.

     There certainly are issues.  Not only the ordering, but the directory
     cookies.  You can change the cookie verifier but I don't know what
     clients do in that case (even for v3).  I suspect that not many
     can continue the READDIR, and if rearrangement is allowed I doubt 
     any can.  I think this is a case of "Doctor, it hurts when I do
     that"  "Don't do that".  If directory ordering changes on a migration,
     READDIR will at least be interrupted.

     The spec defines a migration mechanism for the client point of 
     view.  It is up to the server to make the source and destination
     sufficiently similar.  One obvious way to do that is for the 
     destination to be image copy the source.  There may be other
     solutions, but allowing the client to cope with any transition
     where the source and destination are "logically" the same for
     some definition of "logically" is not possible in v4.0.  Maybe
     in v4.1.  But for now, I don't see the spec tackling this.

Issue 142: Clarify the server's ability to change its lease period at 
server restart and the fact that the server must have a grace period that 
is at least as long as the lease period used in the server's prior 
instantiation. Clients should therefore obtain the lease period at 
each server restart. 

     I suppose we can write a paragraph for this, especially since it
     also is an issue for migration.

Issue 143: Migration identification; relies on stateid; is it important 
for filesystems for which there is no state held.

     It does not rely on stateid.  If you have no state, you still
     get a MOVED error when you reference that filesystem.  It's just 
     that if you don't have state, you don't need a timely indication,
     before you actually reference the fs. 

Issue 144: Need further definition of clientaddr4 struct members. 
Specifically r_netid and r_addr from: 

     Don't understand what is required here. 

Issue 145: Callback identification relies on the client identifying 
itself and the use of a separate communication channel. This approach 
can result in callback paths that are not functional when a NAT or 
firewall is in place between the client and server. Should the RPC 
channel established by the client be used by the server for callbacks? 

     This has been discussed a lot.  I think it is pretty late in the
     day to be making such a change even as option.  

Issue 147: The RFC is restrictive in its definition of what occurs for an 
OPEN(create/UNCHECKED) with a size specified as non-zero. Should this be 
changed to state that if the server can support the truncation to a 
non-zero length on OPEN(create) that the operation is allowed. Text is 
located in OPEN DESCRIPTION. 

     I see no reason to make this change.  It's just another thing that
     some servers might be able to do and others not.  Is there a big demand
     for this?

Issue 147A: Clarify, for the ACCESS operation, if the server should return 
"supported" for the ACCESS4_LOOKUP and ACCESS4_EXECUTE bits when the 
client requests them for object types where they are defined to have 
"no meaning". 

     Why bother?  They don't have a meaning so nobody should be looking
     at them so who cares what the server sets them to?

---

From: [email-masked]
Date: Thu, 18 Jul 2002 11:49:22 -0700

Thanks for the gargantuan issue review.

I think I will respond using separate emails for separate issues.

You listed Issue 94 as not worth holding the spec up for.

>>>>> "Dave" == Dave Noveck <Noveck> writes:

    Dave> Issue 94: Clarify the responsibility of the client to "clean
    Dave> up" state at the server for operations that may be aborted
    Dave> at the client. For example, if an application is awaiting a
    Dave> lock request and that application is killed at the client
    Dave> (Unix ^C), the client is responsible for ensuring that lock
    Dave> request in progress with the server is accounted for. This
    Dave> means that the client may release the application so that it
    Dave> may be stopped before receiving a response from the server
    Dave> for the lock request but an appropriate UNLOCK and CLOSE
    Dave> must be done. There are other places where this type of
    Dave> state clean up is required of the client.

    Dave>      I think going in this direction is a mistake.  It gets
    Dave>      way to much into the details of client implementations.
    Dave>      The principle is that if you do something that creates
    Dave>      state you have to undo that.  The whole thing about
    Dave>      applications and control-C is solely within the purview
    Dave>      of the client implementation.

I agree that the spec should not get too far into implementation
details (for some definition of "too far").  

On the other hand, this issue (what the client should do if the user
hits ^C while waiting for a lock) has been a major, recurring headache
for prior versions of NFS.  If there is verbage we can put in the spec
so that clients are more likely to get this right, I see that as a
Very Good Thing.

Sigh, I guess this means I'm signing up to write the verbage, eh?

mike

---

From: [email-masked]
Date: Thu, 18 Jul 2002 11:50:04 -0700

Issue 105 is in your "should not hold up the spec" list.

>>>>> "Dave" == Dave Noveck <Noveck> writes:

    Dave> Issue 105: Clarify the use of CLOSE (with delegation
    Dave> stateid) as a means for the client to return a delegation to
    Dave> the server. Is the client required to CLOSE for both the
    Dave> original OPEN and the delegation stateid or is the
    Dave> delegation stateid CLOSEd when the CLOSE is done for the
    Dave> OPEN stateid? Or should the client CLOSE with the stateid
    Dave> returned from OPEN as well as DELEGRETURN with the
    Dave> delegation stateid?

    Dave>      The last of these is correct.  I don't see the problem.
    Dave>      Delegations are returned with DELEGRETURN and OPEN's
    Dave>      are bracketed with CLOSE's.  What more needs to be
    Dave>      said?

If this is the issue I'm thinking of, there has been some difference
of opinion as to how long the client can hold onto the delegation.
In particular, can the client 

    CLOSE the original reference

and then

    OPEN using CLAIM_DELEGATE_CUR with the delegation from the
      original reference

It seems wrong to me that the client keeps its delegation even after
issuing a CLOSE on the file.  I seem to be in the minority on this,
and I can live with that behavior being allowed, but it does seem
important for interoperability that the spec clarify this.

mike

---

From: Mike Kupfer <[email-masked]>
Date: Thu, 18 Jul 2002 11:52:04 -0700

>>>>> "Dave" == Dave Noveck <Noveck> writes:

    Dave> Issue 99: Section 9.7 says not to do negative LOOKUP
    Dave> caching. But if we're doing READDIR caching, shouldn't it be
    Dave> okay to have a negative LOOKUP cache entry, as long as it
    Dave> expires when the READDIR cache expires?

    Dave>      I think the intention was for OPEN's.  In the case of
    Dave>      positive name caching, it is clear you can't use this
    Dave>      for OPEN's, but for non-existence you might be tempted
    Dave>      to avoid the OPEN attempt and I don't think that would
    Dave>      be right.  Anyone else have opinions about this?

I don't have a problem with prohibiting negative caching, but I think
the spec is ambiguous and could be misinterpreted.

The start of Section 9.7 says

   The results of LOOKUP and READDIR operations may be cached to avoid
   the cost of subsequent LOOKUP operations.

That could be taken to mean that the READDIR cache can be used to
avoid LOOKUP calls that the client "knows" will fail.

I suppose that this sentence, combined with the first bullet item,
implies that if the entry is not in the READDIR cache, the client
should issue a LOOKUP anyway.  But I'd rather have the spec say that
explicitly, rather than forcing the client implementor to notice the
implication.

mike