Subject: [nfsv4] location_info

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 26 Aug 2005 16:30:12 -0400

This is my current take on location_info.  Thanks to Jon Haswell
for some excellent suggestions.  I've tried to incorporate his
suggested function although I've changed the mechanism a bit
compared to previous discussion.  The basic changes are 
designed to be open to larger and more complicated sets of 
replicas.  I may have a ton of sets of replicas and some may
have multiple servers clustered while some of those may have
multiple paths.  Given the potentially large set of replicas
I have tried to keep things as compact as possible without 
unduly restricting what the server can tell the clients.
Also, since reliability may be a prime consideration in selecting
a server in the first place, the client may need to know when
selecting server A, whether there are other servers to which it
might switch with minimal disruption.  This means that a cleint
might want information on the compatibility of any pair of 
servers rather than just the compatibility of one server with the
current one, and I've tried to incorporate that information.

     The location_info attribute is intended as a more functional
     replacement for fs_locations which will continue to exist and be
     supported.  Clients which need the additional information provided
     by this attribute will interrogate it and get the information from
     servers that support it.  When the server does not support
     location_info, fs_locations can be used to get a subset of the
     information.  A server which supports location_info SHOULD support
     fs_locations as well.

     There are several sorts of additional information present in
     location_info, that aren't available in fs_locations:

     o    Attribute continuity information to allow a client to select a
          location which meets the transparency requirements of the
          applications accessing the data and to take advantage of
          optimizations that server guarantees as to attribute
          continuity may provide (e.g. change attribute).

     o    Filesystem identity information which indicates when multiple
          replicas, from the clients point of view, correspond to the
          same target filesystem, allowing them to be used
          interchangeably, without disruption, as multiple paths to the
          same thing.

     o    Information which will bear on the suitability of various
          replicas, depending on the use that the client intends.  For
          example, many applications need an absolutely up-to-date copy
          (e.g. those that write), while others may only need access to
          the most up-to-date copy reasonably available.

     o    Server-derived preference information for replicas, which can
          be used to implement load-balancing while giving the client
          the entire fs list to be used in case the primary fails.

     Attribute continuity and filesystem identity information define a
     number of identity relations among the various filesystem replicas.
     Most often, the relevant question for the client will be whether a
     given replica is identical-with/continuous-to the current one in a
     given respect but the information should be available also as to
     whether two other replicas match in that respect as well.

     The way in which such pairwise filesystem comparisons is relatively
     compactly encoded is to associate with each replica a 32-bit
     integer, location id.  The location_info attribute then contains
     for each of the identity relations among replicas a 32-bit mask.
     If that mask, when anded with the location ids of the two replicas,
     results in values which are identical, then the two replicas are 
     defined as belonging to the corresponding identity relation.  This 
     scheme allows the server to accommodate relatively large sets of 
     replicas distinct according to a given criteria without requiring 
     large amounts of data to be sent for each replica.

     Server-specified preference information is also provided in a
     fashion that allows a number of different relations (in this case
     order relations) in a compact way.  In this case each
     location4_server structure contains a 32-bit priority word which
can
     be broken into fields devoted to these relations in any way the
     server wishes.  The location4_info structure contains a set of
     32-bit masks, one for each relation.  Two replicas can be compared
     via that relation by anding the corresponding mask with the
     priority word for each replica and comparing the results.

     The location_info attribute consists of a root pathname (just like
     fs_locations), together with an array of location4_item structures.

          struct location4_server {
                  uint32_t        priority;
                  uint32_t        flags;
                  uint32_t        location_id;
                  uint32_t        currency;
                  utf8str_cis     server;
          };

          struct location4_item {
                  location4_server entries<>;
                  pathname4       rootpath;
          };

          struct location4_info {
                  pathname4       fs_root;
                  location4_item  items<>;
                  uint32_t        fileid_keep_mask;
                  uint32_t        change_cont_mask;
                  uint32_t        data_same_mask;
                  uint32_t        same_fh_mask;
                  uint32_t        same_state_mask;
                  uint32_t        same_fs_mask;
                  uint32_t        time_to_switch;
                  uint32_t        valid_for;
                  uint32_t        read_rank_mask;
                  uint32_t        read_order_mask;
                  uint32_t        write_rank_mask;
                  uint32_t        write_order_mask;

          };

     The location_info attribute is structured similarly to the
     fs_locations attribute.  A top-level structure (fs_locations4 or
     locations4_info) contains the entire attribute including the root
     pathname of the fs and an array of lower-level structures that
     define replicas that share a common root path on their respective
     servers.  Those lower-level structures in turn (fs_locations4 or
     location4_item) contain a specific pathname and information on one
     or more individual server replicas.  For that last lowest-level
     information, fs_locations has a server name in the form of
     utf8str_cis, while location_info has a location4_server structure
     that contains per-server-replica information in addition to the
     server name.

     The location4_server structure consists of the following items:

     o    The priority word used to implement server-specified ordering
          relations among replicas.  These relations are intended to be
          used to select a replica when migration (including a referral)
          occurs, when a server appears to be down, when the server
          direct the client to find a new replica (see LIF_GOING) and
          optionally when a new filesystem is first entered.  See the
          location4_info fields read_order, read_rank, write_order, and
          write_rank for details of the ordering relations.

     o    A word of flags providing information about this
          replica/target.  These flags are defined below.

     o    An indication of file system up-to-date-ness (currency) in
          terms of approximate seconds before the present with zero
          indicating the real thing or a reliably up-to-date copy, and
          positive values indicating less-then- current copies of that
          approximate age.

     o    A location id for the replica, to be used together with masks
          in the location4_info structure to determine whether that
          replica matches other in various respects, as described above.
          When two location id's are identical than access to the
          corresponding replicas are defined as identical in all
          respects.  They access the same filesystem with the same
          filehandles and share v4 file state.  Further, multiple
          connections to the two replicas may be done as part of the
          same session.  Two such replicas will share a common root path
          and are best presented within two location4_server entries in
a
          common location4_item.  These replicas should have identical
          values for the currency field although the flags and priority
          fields may be different.

     o    The server string.  For the case of the replica currently
          being accessed (via GETATTR), a null string may be used to
          indicate the current address.

     The flags field has the following bits defined:

     o    LIF_MULTI_FS indicates that when a migration even occurs from
          the current replica to this one, the replacement may consist
          of multiple filesystems.  In this case, the client has to be
          prepared for the possibility that objects on the same fs
          before migration will be on different ones after.  Note that
          LIF_MULTI_FS is not incompatible with the two filesystems
          agreeing fileid-keep mask since, if one has a set of fileid's
          that are unique within an fs, each subset assigned to a
          smaller fs after migration would not have any conflicts
          internal to that fs.

     o    LIF_WRITABLE indicates that the this fs target is writable.

     o    LIF_CACHE indicates that the server is a cached copy and that
          the latency measured by the client may differ radically
          between recently accessed data and that not recently accessed.

     o    LIF_CUR_REQ indicates that this replica is the one on which
          the request is being made.  Only a single server entry may
          have this flag set and in the case of a referral, no entry
          will have it.

     o    LIF_ABSENT indicates that this entry corresponds an absent
          filesystem replica.  It can only be set if LIF_CUR_REQ is set.
          When both such bits are set it indicates that replica is not
          usable but that the information in the entry can be used to
          determine the sorts of continuity available when switching
          from this replica to other possible replicas.

     o    LIF_GOING indicates that a replica while still available
          should not be used further.  The client, if using it, should
          make an orderly transfer within a specified period (see the
          location4_info field time_to_switch).

     The location4_item structure, analogous to a fs_location4
     structure, specifies the root pathname all used by an array of
     server replica entries.

     The location4_info structure, encoding the location4_info attribute
     contains the following:

     o    The fs_root field which contains the pathname of the root of
          the current filesystem on the current server, just as it does
          the fs_locations4 structure.

     o    An array of location4_item structures, which contain
          information about replicas of the current filesystem.  Where
          the current filesystem is actually present, or has been
          present, i.e. this is not a referral situation, one of the
          location4_item structure will contain a location4_server for
          the current server.  This structure will have LIF_ABSENT set
          if the current filesystem is absent, i.e.  normal access to it
          will return NFS4ERR_MOVED.

     o    The fileid-keep mask indicates, in combination with the
          appropriate location ids that fileids will not change (i.e.
          hey will be reliably maintained with no lack of continuity)
          across a migration event between any pair of replicas.  This
          allows migration to safely proceed without any chance that
          application that depend on fileids will be impacted.

     o    The change-cont mask indicates, in combination with the
          appropriate location ids that the change attribute is
          continuous across a migration event between the server within
          any pair of replicas.  In other words if the change attribute
          has a given value before the migration event, then it will
          have that same value after, unless there has been an
          intervening change to the file.  This information useful after
          a migration event, in avoiding any need to refetch change
          information or any requirement to needlessly flush cached data
          because of a lack of reliable change information.

     o    The data-same mask indicates, in combination with the
          appropriate location ids whether any pair of filesystem
          replicas are guaranteed to consist of the same data.  When
          guarantee is not provided, the client (and associated
          applications) may see slightly different data before and after
          a migration event.

     o    The same-fh mask indicates, in combination with the
          appropriate location ids whether two replicas will have the
          same fh's for correspoing objects.  When this is true,
          persistent filehandles may be used across a migration event.

     o    The same-state mask indicates, in combination with the
          appropriate location ids whether two replicas will have the
          same state environment.  This does not necessarily mean that
          when doing performing migration the client will not have to
          reclaim state.  However it does mean that the client may
          proceed using his current clientid just as if there were no
          migration event only reclaim state when an
          NFS4ERR_STALE_CLIENTID error is received.

     o    The same-fs mask indicates, in combination with the
          appropriate location ids whether two replicas in fact
          designate the same filesystem in all respects.  If so, any
          action taken on one is immediately on the other and the client
          can consider them as effectively the same thing.  The same-fs
          mask must include all bits in the same-fh mask, the change-
          cont mask, same-state mask, and the data-same mask, but may
          include bits not in any of those.  When two replicas' location
          ids match on all the bits within the same-fs mask, but are not
          identical, the client using sessions will establish separate
          sessions to each which together share a common clientid.

     o    The time_to_switch field gives a number of seconds by which a
          client using a replica marked LIF_GOING is to cease using it.
          Normally, an NFS4ERR_MOVED would be expected after this period
          of time.  The time_to_switch provides an opportunity for an
          orderly transfer of state and a greater continuity of service.

     o    The valid_for field specifies a time for which it is
          reasonable for a client to use the location_info attribute
          without refetch.  It does not provide a guarantee of validity
          since servers can unexpectedly go out of service or become
          inaccessible for any number of reasons.  Clients are well-
          advised to refetch this information for actively accessed
          filesystem at every valid_for seconds.  This is particularly
          important when filesystem replica may go out of service in a
          controlled way using the LIF_GOING flag to communicate an
          ongoing change.  The server should set valid_for to a value
          which allows well-behaved clients to notice the LIF_GOING flag
          and make an orderly switch before the time_to_switch expires.

     o    The read-rank, read-order, write-rank, and write-order masks
          are used, together with the priority words of various replicas
          to order the replicas according to the server's preference.
          See the discussion below for the interaction of rank, order,
          and the clients own preferences and needs.  Read-rank and
          read-order are used to direct clients which only need read
          access while write-rank and write-order and used to control
          access to direct clients that require some degree of write
          access to the filesystem.  Server's which which to export
          replica order preferences that do not distinguish read and 
          write access can simply present a location4_info in which  
          the read-rank and write-rank mask, and the read-order and 
          write-order mask are equal.

     Depending on the potential need for write access, one of the pairs
     of rank and order masks is used, together with replica priority
     words, is used to determine a rank and an order for each replica
     under consideration.  The rank is obtained by anding the selected
     rank ask with the priority and the order is obtained similarly by
     anding the selected order mask with the priority.  The resulting
     rank and order are compared as described below with lower always
     being better.

     Rank is sued to express a strict server-imposed ordering on
     clients.  Clients should attempt to use all replicas with a given
     before they use one with a higher rank.  Only if all of those
     servers are unavailable or unsuitable should the client proceed to
     servers of a higher rank.  Clients may use such things as currency,
     measured latency or the availability of multiple paths only if the
     declared attributes of the replicas make the replica unsuitable for
     desired purpose at all, rather than in finding a replica which is
     "better".  Thus if a client require up-to-date data it may ignore
     servers whose currency value is non zero.  Similarly, if a server
     has a latency so large that it can be considered down, then
     replicas of other ranks should be considered.

     Within a rank, the order value is used to specify the server's
     preference to guide the clients selection when the client's own
     preferences are not controlling.  If replicas are approximately
     equal in all respects, clients should defer to the order specified
     by the server.  When clients look at server latency as part of
     their selection, they are free to use this criterion but it is
     suggested that when latency differences are not significant, the
     server-specified order should guide selection.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 26 Aug 2005 14:19:53 -0700

>           indicate the current address.

One of the things I've wanted to see is for the server to be able to
advertise that it supports access via different transport mechanisms,
and to be able to list those transports to any given client.

IOW: As a client, I'd like to be able to figure out not only on which
server the replicas are hosted, but also whether or not I can use a
faster transport mechanism, such as RDMA, say, in order to access the
data. I would also be interested in figuring out whether or not a
particular replica might support a feature like parallel read/write
access via pNFS.

Do you have any thoughts about how to do this?

Cheers,
  Trond

---

From: Jon Haswell <[email-masked]>
Date: Tue, 30 Aug 2005 13:03:18 -0700

>From my perspective this all looks very good. When I wrote my note I wasn't
sure that we were ready for something as all encompassing as this proposal,
but I'd be more than happy if this got accepted.

The one small things (there's always one isn't there) that I might suggest
is missing is actually being able to specify some form of version string
for each replica instance. I would suggest a string so that the server
could choose to put in a date/time at which a snapshot was taken for
example, or an incrementing version number as a replica is updated or
whatever it chooses. I see this primarily as a debug tool so that we can
include in log messages some human readable string that can easily tell us
that this client was getting stale data because server x was working from
an old replica.

Thanks

Jon Haswell
IBM Research
Tel 408 927 2074 Tie 457 2074
Fax 408 927 3030 Tie 457 3030