Subject: RE: [nfsv4] location_info

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 30 Aug 2005 16:59:56 -0400

>From your description, it doesn't seem that this would be involved
in the client choosing among alternatives since the client is not
really supposed to interpret it.  Since this is for debugging
once the cleint has selected something, maybe this would make
sense as an fs attribute interrogated by the client on the target
replica and then logged somewhere for later debugging.  How about 
fs_provenance? 

-----Original Message-----
From: Jon Haswell [mailto:[email-masked]] 
Sent: Tuesday, August 30, 2005 4:03 PM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] location_info

>From my perspective this all looks very good. When I wrote my note I
wasn't
sure that we were ready for something as all encompassing as this
proposal,
but I'd be more than happy if this got accepted.

The one small things (there's always one isn't there) that I might
suggest
is missing is actually being able to specify some form of version string
for each replica instance. I would suggest a string so that the server
could choose to put in a date/time at which a snapshot was taken for
example, or an incrementing version number as a replica is updated or
whatever it chooses. I see this primarily as a debug tool so that we can
include in log messages some human readable string that can easily tell
us
that this client was getting stale data because server x was working
from
an old replica.

Thanks

Jon Haswell
IBM Research
Tel 408 927 2074 Tie 457 2074
Fax 408 927 3030 Tie 457 3030

[email-masked] wrote on 08/26/2005 01:30:12 PM:

> This is my current take on location_info.  Thanks to Jon Haswell
> for some excellent suggestions.  I've tried to incorporate his
> suggested function although I've changed the mechanism a bit
> compared to previous discussion.  The basic changes are
> designed to be open to larger and more complicated sets of
> replicas.  I may have a ton of sets of replicas and some may
> have multiple servers clustered while some of those may have
> multiple paths.  Given the potentially large set of replicas
> I have tried to keep things as compact as possible without
> unduly restricting what the server can tell the clients.
> Also, since reliability may be a prime consideration in selecting
> a server in the first place, the client may need to know when
> selecting server A, whether there are other servers to which it
> might switch with minimal disruption.  This means that a cleint
> might want information on the compatibility of any pair of
> servers rather than just the compatibility of one server with the
> current one, and I've tried to incorporate that information.
>
>      The location_info attribute is intended as a more functional
>      replacement for fs_locations which will continue to exist and be
>      supported.  Clients which need the additional information
provided
>      by this attribute will interrogate it and get the information
from
>      servers that support it.  When the server does not support
>      location_info, fs_locations can be used to get a subset of the
>      information.  A server which supports location_info SHOULD
support
>      fs_locations as well.
>
>      There are several sorts of additional information present in
>      location_info, that aren't available in fs_locations:
>
>      o    Attribute continuity information to allow a client to select
a
>           location which meets the transparency requirements of the
>           applications accessing the data and to take advantage of
>           optimizations that server guarantees as to attribute
>           continuity may provide (e.g. change attribute).
>
>      o    Filesystem identity information which indicates when
multiple
>           replicas, from the clients point of view, correspond to the
>           same target filesystem, allowing them to be used
>           interchangeably, without disruption, as multiple paths to
the
>           same thing.
>
>      o    Information which will bear on the suitability of various
>           replicas, depending on the use that the client intends.  For
>           example, many applications need an absolutely up-to-date
copy
>           (e.g. those that write), while others may only need access
to
>           the most up-to-date copy reasonably available.
>
>      o    Server-derived preference information for replicas, which
can
>           be used to implement load-balancing while giving the client
>           the entire fs list to be used in case the primary fails.
>
>      Attribute continuity and filesystem identity information define a
>      number of identity relations among the various filesystem
replicas.
>      Most often, the relevant question for the client will be whether
a
>      given replica is identical-with/continuous-to the current one in
a
>      given respect but the information should be available also as to
>      whether two other replicas match in that respect as well.
>
>      The way in which such pairwise filesystem comparisons is
relatively
>      compactly encoded is to associate with each replica a 32-bit
>      integer, location id.  The location_info attribute then contains
>      for each of the identity relations among replicas a 32-bit mask.
>      If that mask, when anded with the location ids of the two
replicas,
>      results in values which are identical, then the two replicas are
>      defined as belonging to the corresponding identity relation.
This
>      scheme allows the server to accommodate relatively large sets of
>      replicas distinct according to a given criteria without requiring
>      large amounts of data to be sent for each replica.
>
>      Server-specified preference information is also provided in a
>      fashion that allows a number of different relations (in this case
>      order relations) in a compact way.  In this case each
>      location4_server structure contains a 32-bit priority word which
> can
>      be broken into fields devoted to these relations in any way the
>      server wishes.  The location4_info structure contains a set of
>      32-bit masks, one for each relation.  Two replicas can be
compared
>      via that relation by anding the corresponding mask with the
>      priority word for each replica and comparing the results.
>
>      The location_info attribute consists of a root pathname (just
like
>      fs_locations), together with an array of location4_item
structures.
>
>
>           struct location4_server {
>                   uint32_t        priority;
>                   uint32_t        flags;
>                   uint32_t        location_id;
>                   uint32_t        currency;
>                   utf8str_cis     server;
>           };
>
>           struct location4_item {
>                   location4_server entries<>;
>                   pathname4       rootpath;
>           };
>
>           struct location4_info {
>                   pathname4       fs_root;
>                   location4_item  items<>;
>                   uint32_t        fileid_keep_mask;
>                   uint32_t        change_cont_mask;
>                   uint32_t        data_same_mask;
>                   uint32_t        same_fh_mask;
>                   uint32_t        same_state_mask;
>                   uint32_t        same_fs_mask;
>                   uint32_t        time_to_switch;
>                   uint32_t        valid_for;
>                   uint32_t        read_rank_mask;
>                   uint32_t        read_order_mask;
>                   uint32_t        write_rank_mask;
>                   uint32_t        write_order_mask;
>
>           };
>
>      The location_info attribute is structured similarly to the
>      fs_locations attribute.  A top-level structure (fs_locations4 or
>      locations4_info) contains the entire attribute including the root
>      pathname of the fs and an array of lower-level structures that
>      define replicas that share a common root path on their respective
>      servers.  Those lower-level structures in turn (fs_locations4 or
>      location4_item) contain a specific pathname and information on
one
>      or more individual server replicas.  For that last lowest-level
>      information, fs_locations has a server name in the form of
>      utf8str_cis, while location_info has a location4_server structure
>      that contains per-server-replica information in addition to the
>      server name.
>
>      The location4_server structure consists of the following items:
>
>      o    The priority word used to implement server-specified
ordering
>           relations among replicas.  These relations are intended to
be
>           used to select a replica when migration (including a
referral)
>           occurs, when a server appears to be down, when the server
>           direct the client to find a new replica (see LIF_GOING) and
>           optionally when a new filesystem is first entered.  See the
>           location4_info fields read_order, read_rank, write_order,
and
>           write_rank for details of the ordering relations.
>
>      o    A word of flags providing information about this
>           replica/target.  These flags are defined below.
>
>      o    An indication of file system up-to-date-ness (currency) in
>           terms of approximate seconds before the present with zero
>           indicating the real thing or a reliably up-to-date copy, and
>           positive values indicating less-then- current copies of that
>           approximate age.
>
>      o    A location id for the replica, to be used together with
masks
>           in the location4_info structure to determine whether that
>           replica matches other in various respects, as described
above.
>           When two location id's are identical than access to the
>           corresponding replicas are defined as identical in all
>           respects.  They access the same filesystem with the same
>           filehandles and share v4 file state.  Further, multiple
>           connections to the two replicas may be done as part of the
>           same session.  Two such replicas will share a common root
path
>           and are best presented within two location4_server entries
in
> a
>           common location4_item.  These replicas should have identical
>           values for the currency field although the flags and
priority
>           fields may be different.
>
>      o    The server string.  For the case of the replica currently
>           being accessed (via GETATTR), a null string may be used to
>           indicate the current address.
>
>      The flags field has the following bits defined:
>
>      o    LIF_MULTI_FS indicates that when a migration even occurs
from
>           the current replica to this one, the replacement may consist
>           of multiple filesystems.  In this case, the client has to be
>           prepared for the possibility that objects on the same fs
>           before migration will be on different ones after.  Note that
>           LIF_MULTI_FS is not incompatible with the two filesystems
>           agreeing fileid-keep mask since, if one has a set of
fileid's
>           that are unique within an fs, each subset assigned to a
>           smaller fs after migration would not have any conflicts
>           internal to that fs.
>
>      o    LIF_WRITABLE indicates that the this fs target is writable.
>
>      o    LIF_CACHE indicates that the server is a cached copy and
that
>           the latency measured by the client may differ radically
>           between recently accessed data and that not recently
accessed.
>
>      o    LIF_CUR_REQ indicates that this replica is the one on which
>           the request is being made.  Only a single server entry may
>           have this flag set and in the case of a referral, no entry
>           will have it.
>
>      o    LIF_ABSENT indicates that this entry corresponds an absent
>           filesystem replica.  It can only be set if LIF_CUR_REQ is
set.
>           When both such bits are set it indicates that replica is not
>           usable but that the information in the entry can be used to
>           determine the sorts of continuity available when switching
>           from this replica to other possible replicas.
>
>      o    LIF_GOING indicates that a replica while still available
>           should not be used further.  The client, if using it, should
>           make an orderly transfer within a specified period (see the
>           location4_info field time_to_switch).
>
>      The location4_item structure, analogous to a fs_location4
>      structure, specifies the root pathname all used by an array of
>      server replica entries.
>
>      The location4_info structure, encoding the location4_info
attribute
>      contains the following:
>
>      o    The fs_root field which contains the pathname of the root of
>           the current filesystem on the current server, just as it
does
>           the fs_locations4 structure.
>
>      o    An array of location4_item structures, which contain
>           information about replicas of the current filesystem.  Where
>           the current filesystem is actually present, or has been
>           present, i.e. this is not a referral situation, one of the
>           location4_item structure will contain a location4_server for
>           the current server.  This structure will have LIF_ABSENT set
>           if the current filesystem is absent, i.e.  normal access to
it
>           will return NFS4ERR_MOVED.
>
>      o    The fileid-keep mask indicates, in combination with the
>           appropriate location ids that fileids will not change (i.e.
>           hey will be reliably maintained with no lack of continuity)
>           across a migration event between any pair of replicas.  This
>           allows migration to safely proceed without any chance that
>           application that depend on fileids will be impacted.
>
>      o    The change-cont mask indicates, in combination with the
>           appropriate location ids that the change attribute is
>           continuous across a migration event between the server
within
>           any pair of replicas.  In other words if the change
attribute
>           has a given value before the migration event, then it will
>           have that same value after, unless there has been an
>           intervening change to the file.  This information useful
after
>           a migration event, in avoiding any need to refetch change
>           information or any requirement to needlessly flush cached
data
>           because of a lack of reliable change information.
>
>      o    The data-same mask indicates, in combination with the
>           appropriate location ids whether any pair of filesystem
>           replicas are guaranteed to consist of the same data.  When
>           guarantee is not provided, the client (and associated
>           applications) may see slightly different data before and
after
>           a migration event.
>
>      o    The same-fh mask indicates, in combination with the
>           appropriate location ids whether two replicas will have the
>           same fh's for correspoing objects.  When this is true,
>           persistent filehandles may be used across a migration event.
>
>      o    The same-state mask indicates, in combination with the
>           appropriate location ids whether two replicas will have the
>           same state environment.  This does not necessarily mean that
>           when doing performing migration the client will not have to
>           reclaim state.  However it does mean that the client may
>           proceed using his current clientid just as if there were no
>           migration event only reclaim state when an
>           NFS4ERR_STALE_CLIENTID error is received.
>
>      o    The same-fs mask indicates, in combination with the
>           appropriate location ids whether two replicas in fact
>           designate the same filesystem in all respects.  If so, any
>           action taken on one is immediately on the other and the
client
>           can consider them as effectively the same thing.  The
same-fs
>           mask must include all bits in the same-fh mask, the change-
>           cont mask, same-state mask, and the data-same mask, but may
>           include bits not in any of those.  When two replicas'
location
>           ids match on all the bits within the same-fs mask, but are
not
>           identical, the client using sessions will establish separate
>           sessions to each which together share a common clientid.
>
>      o    The time_to_switch field gives a number of seconds by which
a
>           client using a replica marked LIF_GOING is to cease using
it.
>           Normally, an NFS4ERR_MOVED would be expected after this
period
>           of time.  The time_to_switch provides an opportunity for an
>           orderly transfer of state and a greater continuity of
service.
>
>      o    The valid_for field specifies a time for which it is
>           reasonable for a client to use the location_info attribute
>           without refetch.  It does not provide a guarantee of
validity
>           since servers can unexpectedly go out of service or become
>           inaccessible for any number of reasons.  Clients are well-
>           advised to refetch this information for actively accessed
>           filesystem at every valid_for seconds.  This is particularly
>           important when filesystem replica may go out of service in a
>           controlled way using the LIF_GOING flag to communicate an
>           ongoing change.  The server should set valid_for to a value
>           which allows well-behaved clients to notice the LIF_GOING
flag
>           and make an orderly switch before the time_to_switch
expires.
>
>      o    The read-rank, read-order, write-rank, and write-order masks
>           are used, together with the priority words of various
replicas
>           to order the replicas according to the server's preference.
>           See the discussion below for the interaction of rank, order,
>           and the clients own preferences and needs.  Read-rank and
>           read-order are used to direct clients which only need read
>           access while write-rank and write-order and used to control
>           access to direct clients that require some degree of write
>           access to the filesystem.  Server's which which to export
>           replica order preferences that do not distinguish read and
>           write access can simply present a location4_info in which
>           the read-rank and write-rank mask, and the read-order and
>           write-order mask are equal.
>
>      Depending on the potential need for write access, one of the
pairs
>      of rank and order masks is used, together with replica priority
>      words, is used to determine a rank and an order for each replica
>      under consideration.  The rank is obtained by anding the selected
>      rank ask with the priority and the order is obtained similarly by
>      anding the selected order mask with the priority.  The resulting
>      rank and order are compared as described below with lower always
>      being better.
>
>      Rank is sued to express a strict server-imposed ordering on
>      clients.  Clients should attempt to use all replicas with a given
>      before they use one with a higher rank.  Only if all of those
>      servers are unavailable or unsuitable should the client proceed
to
>      servers of a higher rank.  Clients may use such things as
currency,
>      measured latency or the availability of multiple paths only if
the
>      declared attributes of the replicas make the replica unsuitable
for
>      desired purpose at all, rather than in finding a replica which is
>      "better".  Thus if a client require up-to-date data it may ignore
>      servers whose currency value is non zero.  Similarly, if a server
>      has a latency so large that it can be considered down, then
>      replicas of other ranks should be considered.
>
>      Within a rank, the order value is used to specify the server's
>      preference to guide the clients selection when the client's own
>      preferences are not controlling.  If replicas are approximately
>      equal in all respects, clients should defer to the order
specified
>      by the server.  When clients look at server latency as part of
>      their selection, they are free to use this criterion but it is
>      suggested that when latency differences are not significant, the
>      server-specified order should guide selection.
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Jon Haswell <[email-masked]>
Date: Tue, 30 Aug 2005 14:12:37 -0700

Yes, that would certainly work, my only thought for putting it in the
location_info field would be to avoid creating just one more attribute.
Your suggested names also is kind of fun.

Thanks

Jon Haswell
IBM Research

---

From: Ted Anderson <[email-masked]>
Date: Wed, 31 Aug 2005 10:45:41 -0400

> Your suggested names also is kind of fun.

One valuable use for the replica version number does relate to replica
selection.  A feature of DFS is that even though a client may
unavoidably deliver inconsistent data to an application when
transitioning from an older replica to a newer one, the client
guarantees never to provide data from an older replica.  This means the
age of data is always monotonic increasing.

To implement this the client queries the replica server and compares
replica (in the DFS case: fileset) version numbers and ignores replicas
whose version number is older than the latest version it has ever
fetched a file from.

In the NFSv4 case, this replica attribute could be part of location_info
or a new per filesystem attribute (e.g. fs_provenence).  The client
would need to be able to make relational comparisons on them, however,
so a number would be better than a string.  In this connection,
certainly, 64 bits would be better than 32.  This would also be
compatible in size with the change attribute and sufficient for a
replica version number, though 96 bits would be necessary to contain an
nfstime4.

Ted

> [email-masked] <[email-masked]> wrote on 08/30/2005 01:59:56 PM:
> 

... [see
http://www1.ietf.org/mail-archive/web/nfsv4/current/msg01940.html] ...

---

From: Carl Burnett <[email-masked]>
Date: Wed, 31 Aug 2005 16:22:55 -0500

I prefer the notion of a new fsid wide attribute, since it can become 
stale shortly after being communicated to clients in locations info. The 
client will want to query it anyway from each candidate server at failover 
time to find a suitable candidate. This also avoids the requirment for a 
server to depend on knowing each peer's version info in order to service a 
locations request.

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

Ted Anderson <[email-masked]> 
Sent by: [email-masked]
08/31/2005 09:45 AM

To
[email-masked]
cc
Jon Haswell/San Jose/IBM@IBMUS, "Noveck, Dave" <[email-masked]>
Subject
Re: [nfsv4] location_info

Ted

... [see