Subject: issue11 (change attribute on client)

From: Mike Eisler <[email-masked]>
Date: Mon, 02 Sep 2002 07:36:01 -0700

Spencer and I have been discussing this privately, and I believe what
follows represents something that is mutually acceptable.

Also, Spencer brought up the issue of clients holding write delegations
that memory map the affected file. This caused me to realize that there are
a large set of interesting consistency issues when one tries to mix
file locking with memory mapping. So I added text to deal with that too.

Issue11: 
The original post for this issue starts here 

    At least two places in the specification say a write
    delegation can modify its cached copy of the "change"
    attribute:
    
       The server will use a CB_GETATTR callback, if the GETATTR attribute
       bits include any of the attributes that a write open delegate may
       modify (size, time_modify, change).
    
    AND:
    
    	    The client returns attrbits and the associated attribute values
    	    only for attributes that it may change (change, time_modify, size).

    How is this possible? The client has no idea how the server is
    constructing the change attribute.

confirm that text added for latest draft is sufficient

--------------------------

I've read the entire specification, and there is still one
issue, in 9.4.3.  Handling of CB_GETATTR:

   For the server, the following steps will be taken when providing a
   write delegation:

   o    On providing a write delegation, the server will cache a copy of
        the change attribute.  Let this value be represented by sc.

   o    The server obtains the change attribute from the client.  Let
        this value be cc.

   o    If the value cc is equal to sc, the file is not modified and the
        server returns the current values for change and time_modify
        (for example) to the client requesting GETATTR.

   o    If the value cc is NOT equal to sc, the file is currently
        modified at the client and most likely will be modified at the
        server at a future time.  The server then uses the current time
        to construct attributes values for change and time_modify and
        returns those values to the requester.

The last bullet item above has a serious problem because
the fragment: "uses the current time to construct
attributes values for change and time_modify" apparently
assumes that the change attribute will be derived from
time_modify. Not at all. The ideal server implementation
uses a unsigned integer that is incremented on each data
change and metadata change (except those metadata changes
that alter time_access an no other metadata). Also, the
server cannot construct change from a high resolution
time_modify even if it wanted to; it has to use
time_metadata if it is going to use any time-based
attribute.

Suggested text follows:

CHANGE:

   at the server.  This means that the server will take the current time
   and apply it to the construction of attributes like change and
   time_modify.

TO:

   at the server.

CHANGE:

   o    The value of the change attribute will be obtained from the
        server and cached.  Let this value be represented by c.

   o    The client will create a value greater than c that will be used
        for communicating modified data is held at the client.  Let this
        value be represented by d.

   o    When the client is queried via CB_GETATTR for the change
        attribute, it checks to see if it holds modified data.  If the
        file is modified, the value d is returned for the change
        attribute value.  If this file is not currently modified, the
        client returns the value c for the change attribute.

TO:

   o    The value of the change attribute will be obtained from the
        server and cached.  Let this value be represented by c.

   o    The client will create a value greater than c that will be used
        for communicating modified data is held at the client.  Let this
        value be represented by d.

   o    When the client is queried via CB_GETATTR for the change
        attribute, it checks to see if it holds modified data.  If the
        file is modified, the value d is returned for the change
        attribute value.  If this file is not currently modified, the
        client returns the value c for the change attribute.

   For simplicity of implementation, the client MAY
   for each CB_GETATTR return the same value d, even if
   between successive CB_GETATTR operations, the client
   again modifies in the file's data or metadata in its
   cache. This because the only requirement is that the
   client be able to indicate to the server that the
   client holds modified data. Therefore, the value of d
   can always be c + 1.

CHANGE:

   While the change attribute is opaque to client in the
   sense that it has no idea what units of time, if any,
   the server is counting change with, it is not opaque
   in that the client has to treat it as an integer, and
   the server has to be able to see the results of the
   client's changes to that integer. Therefore, the server
   MUST encode the change attribute in network order when
   sending it to the client, the client MUST decode it from
   network order to its native order when receiving it,
   and the client MUST encode it network order when sending
   it to the server. For this reason, change is defined
   as an integer, rather than an opaque array of octets.

   For the server, the following steps will be taken when providing a
   write delegation:

   o    On providing a write delegation, the server will cache a copy of
        the change attribute.  Let this value be represented by sc.

   o    The server obtains the change attribute from the client.  Let
        this value be cc.

   o    If the value cc is equal to sc, the file is not modified and the
        server returns the current values for change and time_modify
        (for example) to the client requesting GETATTR.

   o    If the value cc is NOT equal to sc, the file is currently
        modified at the client and most likely will be modified at the
        server at a future time.  The server then uses the current time
        to construct attributes values for change and time_modify and
        returns those values to the requester.

TO:

   While the change attribute is opaque to the client in the sense that it
   has no idea what units of time, if any, the server is counting change
   with, it is not opaque in that the client has to treat it as an
   unsigned integer, and the server has to be able to see the results of the
   client's changes to that integer. Therefore, the server MUST encode
   the change attribute in network order when sending it to the client,
   the client MUST decode it from network order to its native order when
   receiving it, and the client MUST encode it network order when
   sending it to the server. For this reason, change is defined as an
   unsigned integer, rather than an opaque array of octets.

   For the server, the following steps will be taken when providing a
   write delegation:

   o    Upon providing a write delegation, the server will cache a copy of
        the change attribute in the data structure it uses to record
        the delegation.  Let this value be represented by sc.

   o    When a second client sends a GETATTR operation on the same
        file to the server, the server obtains the change attribute from
        the first client.  Let this value be cc.

   o    If the value cc is equal to sc, the file is not modified and the
        server returns the current values for change, time_metadata,
        and time_modify (for example) to the second client.

   o    If the value cc is NOT equal to sc, the file is currently
        modified at the first client and most likely will be modified at the
        server at a future time.  The server then uses its current time
	to construct attribute values for time_metadata
	and time_modify.  A new value of sc, which we
	will call nsc, is computed by the server, such
	that nsc >= sc + 1.  The server then returns the
	constructed time_metadata, time_modify, and nsc
	values to the requester. The server replaces sc
	in the delegation record with nsc. To prevent
	the possibility of time_modify, time_metadata,
	and change from appearing to go backward (which
	would happen if the client holding the delegation
	fails to write its modified data to the server
	before the delegation is revoked or returned),
	the server SHOULD update the file's metadata
	record with the constructed attribute values.
	For reasons of reasonable performance, committing
	the constructed attribute values to stable storage
	is OPTIONAL.

	As discussed earlier in this section, the client
	MAY return the same cc value on subsequent
	CB_GETATTR calls, even if the file was modified
	in the client's cache yet again between successive
	CB_GETATTR calls. Therefore, the server must assume
	that the file has been modified yet again, and MUST
	take care to ensure that the new nsc it constructs
	and returns is greater than the previous nsc it
	returned.  An example implementation's delegation
	record would satisfy this mandate by including a
	boolean field (let us call it "modified") that is
	set to false when the delegation is granted, and
	an sc value set at the time of grant to the change
	attribute value. The modified field would be set to
	true the first time cc != sc, and would stay true
	until the delegation is returned or revoked. The
	processing for constructing nsc, time_modify,
	and time_metadata would use this pseudo code:

          if (!modified) {
            do CB_GETATTR for change and size;

            if (cc != sc)
              modified = TRUE;
          } else {
            do CB_GETATTR for size;
          }

          if (modified) {
             sc = sc + 1;
             time_modify = time_metadata = current_time;
             update sc, time_modify, time_metadata into file's metadata;
          }
          
          return to client (that sent GETATTR) the attributes it requested,
            but make sure size comes from what CB_GETATTR returned. Do
            not update the file's metadata with the client's modified size.

   Instead of doing a CB_GETATTR, the server MAY simply
   recall delegations.

ADD (before 9.7.  Name Caching Delegation): # Note much of this text
comes paraphrased from David Robinson in http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=4drpde%243s7%40engnews2.Eng.Sun.COM 

9.7.  Data and Metadata Caching and Memory Mapped Files

  Some operating environments include the capability
  for an application to map a file's content into the
  application's address space. Each time the application
  accesses a memory location that corresponds to a block
  that has not been loaded into the address space, a page
  fault occurs and the file is read (or if the block does
  not exist in the file, the block is allocated and then
  instantiated in the application's address space).

  As long as each memory mapped access to the file
  requires a page fault, the relevant attributes of the
  file that are used to detect access and modification
  (time_access, time_metadata, time_modify, and change)
  will be updated. However, in many operating environments,
  when page faults are not required these attributes will
  not be updated on reads or updates to the file via memory
  access (regardless whether the file is local file or is
  being access remotely). A client or server MAY fail to
  update attributes of a file that is being accessed via
  memory mapped I/O.  This has several implications:

  o If there is an application on the server that has
    memory mapped a file that a client is also accessing,
    the client may not be able to get a consistent value
    of the change attribute to determine whether its
    cache is stale or not. A server that knows that the
    file is memory mapped could always pessimistically
    return updated values for change so as to force the
    application to always get the most up to date data and
    metadata for the file. However, due to the negative
    performance implications of this, such behavior is
    OPTIONAL.

  o If the memory mapped file is not being modified on
    the server, and instead is just being read by an
    application via the memory mapped interface, the
    client will not see an updated time_access attribute.
    However, in many operating environments, neither will
    any process running on the server. Thus NFS clients
    are at no disadvantage with respect to local processes.

  o If there is another client that is memory mapping the
    file, and if that client is holding a write delegation,
    the same set of issues as discussed in the previous
    two bullet items apply. So, when a server does a
    CB_GETATTR to a file that the client has modified
    in its cache, the response from CB_GETATTR will not
    necessarily be accurate. As discussed earlier, the
    client's obligation is to report that the file has
    been modified since the delegation was granted, not
    whether it has been modified again between successive
    CB_GETATTR calls, and the server MUST assume that any
    file the client has modified in cache has been modified
    again between successive CB_GETATTR calls. Depending
    on the nature of the client's memory management system,
    this weak obligation may not be possible. A client
    MAY return stale information in CB_GETATTR whenever
    the file is memory mapped.

  o The mixture of memory mapping and file locking on the same
    file is problematic. Consider the following scenario,
    where a page size on each client is 8192 bytes.

	 - Client A memory maps first page (8192 bytes) of file X

	 - Client B memory maps first page (8192 bytes) of file X
	 
         - Client A write locks first 4096 bytes

	 - Client B write locks second 4096 bytes

         - Client A, via a STORE instruction modifies part
           of its locked region.

         - Simultaneous to client A, client B issues a
           STORE on part of its locked region.

     Here the challenge is for each client to resynchronize
     to get a correct view of the first page. In many
     operating environments, the virtual memory management
     systems on each client only know a page is modified,
     not that a subset of the page corresponding to the
     respective lock regions has been modified. So it is
     not possible for each client to do the right thing,
     which is to only write to the server that portion of
     the page that is locked.  For example, if client A
     simply writes out the page, and then client B writes
     out the page, client A's data is lost.

     Moreover, if mandatory locking is enabled on
     the file, then we have a different problem. When
     clients A and B issue the STORE instructions, the
     resulting page faults require a record locks on the
     entire page. Each client then tries to extend their
     locked range to the entire page, which results in a
     deadlock. Communicating the NFS4ERR_DEADLOCK error
     to a STORE instruction is difficult at best.

     If a client is locking the entire memory mapped file,
     there is no problem with advisory or mandatory record
     locking, at least until the client unlocks a region
     in the middle of the file.

     Given the above issues the following are permitted:

     - Clients and servers MAY deny memory mapping
       a file they know there are record locks for.

     - Clients and servers MAY deny a record lock on a file
       they know is memory mapped.

     - A client MAY deny memory mapping a file that
       it knows requires mandatory locking for I/O.
       If mandatory locking is enabled after the file
       is opened and mapped, the client MAY deny the
       application further access to its mapped file.

CHANGE: (in 15.2.1.  Operation 3: CB_GETATTR - Get Attributes)

     The client returns attrbits and the associated attribute values
     only for attributes that it may change (change, time_modify, size).

TO: (change is read-only so the above is not quite right; try this):

     The client returns attrmask bits and the associated attribute values
     only for the change attribute, and attributes that it may change
     (time_modify, and size).