Subject: [nfsv4] Locations_info and multiple metadata servers

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 18 Aug 2005 11:12:10 -0400

This is the note I've promised about what I was thinking about a 
possible extension to provide more information about the possible 
replicas or migration targets and how it that might be useful to
provide multi-pathing support and somewhat address some of the
needs of cluster fs implementation in which multiple servers have 
the capability of acting as metadata servers, for a given fs.

Brent Welch wrote:
> I would like to put multiple metadata servers out of scope.
> What worries me is that there are a number of different
> models for how to do it, and that features like the attribute
> you and Marc were discussing don't necessarily apply to all,
> or that don't go far enough.

I see your point, but my perspective is a little different.
I agree that we do not want to take on the task of supporting
every sort of multi-metadata-server arrangement that people 
want (or could conceivably want), or even any fixed set of 
those.  On the other hand, if there are features that make 
sense for v4.1, that do enable some variants of this (even 
if not all or most), then I don't see going out of our way 
to foreclose them, simply because they cross the line into 
multiple-metadata-server territory.  

> In particular, for us we'll have multiple NFSv4 servers that
> export the same file system, and clients can use any of them.
> But, they have to pick one at mount time so you get fairly
> coarse load balancing among them.  "It would be nice" if we
> had a way to disperse clients among the servers, but it
> isn't strictly necessary for pNFS.  Already, by distributing the
> data servers we've offloaded the metadata server a great deal.
> I'm content to stop here, although it would be possible to
> make things even more efficient with yet more protocol.

I agree that multiple metadata servers per se are not something that
we really need to specifically address, but I felt that since we 
have replication which is a means to provide multiple v4 servers 
for a filesystem, it follows that we have, in the pnfs context, a
way to have multiple meta-data servers.  The issue, as it is with 
the non-pnfs case, is giving the client sufficient information to
determine which one he should use, and when he should/may switch 
to a different one, and what the nature of the transition is.  In
other words, we have a way to specify multiple (typically read-only)
replicas and giving the client information to make a proper selection
is a good idea on lots of grounds.  When you have multiple metadata
servers you are basically in adjoining territory: telling the 
client which of a set of replicas he might use and how and when.

RFC 3530 already gives you a set of locations but aside from the 
migration case, in which the server tells you an fs is gone and 
is now somewhere else, there is no real guidance about when you
may or should switch to another server.  You have a list of locations 
on a number of servers.  What other information do you need?

Well, my belief is that the client could find a fair amount of 
additional information pretty helpful, both for initial selection,
and for switching when appropriate.  

So I think we could add a locations_info attribute that contains
what fs_locations does plus some useful additional information.
The fs_locations attribute would remain for those accustomed to 
use it, but it be used less and less, even without formally being
"deprecated".  The locations_info attribute would allow you to 
determine, for each of the locations.

    What the is the server's priority for these?.

    Do two replica represent the exact same data, or they possibly
    slightly different images (snapshots at different times)?

    Which of two replicas (that are different) is more up-to-date?

    Do two replicas have the same fh's (allowing a switch with no
    disruption due to filehandle volatility)?

    Do two replicas agree on fileid's (allowing a switch without 
    breaking applications that use stat and depend on fileid stability
    and uniqueness)?

    Are two replicas going to agree on the defintion of the change
    attribute such that in a transition, you don't have have to
    refetch these values and you can assume that the change attribute
    being the same?

Unless your model is that I will use a replica until it is absolutely
dead and will then pick a different one (at random) and live with 
whatever discontinuities that causes (or rather the applications will
have to live with consequences of the discontinuities) not having this 
information is a problem.				
	       
Clients may want to switch when a server is not performing well, as 
opposed to waiting until death certified by coroner, but to do that 
it needs information on what kind of switch between replicas is involved
and RFC3530 lends itself to multiple interpretations about this issue.
Rather than try to designate a single intepretation as "correct", I 
think we need to recognize that many of these are valuable in various
contexts but the client needs to know which one he is dealing with to
be effective.  If the client has this information he may be able to
switch replicas to deal with poor performance and switch back if it
appears performance is OK.

Carl Burnett suggested an interesting case which I think fits into 
this basic framework which is simply that you may have two replica
designations which are simply different paths to the very same 
realization of a server/fs.  From the viewpoint of the characteristics
mentioned above, they are two replicas that always have the same data,
the same inode numbers, the same filehandles, the same scheme for the
change attribute, and finally the same set of locking data.  Switching 
between such "replicas" is extremely easy and one might then go to the 
next step and simply say that no switching is necessary.  Concurrent 
access to multiple nominal addresses is precisely what the client would
want to do, when a single path is insufficient.

When two IP addresses terminate at the same physical server, the client
may have multiple connections each associated with those distinct
physical address, all part of the same session.

So given that basic context, it would seem that the case of a cluster
fs is one step back from the extreme case (not meant in a pejorative
sense) of multiple paths to the same fs.  Where you have multiple 
servers partaking in access to a clustered filesystem, the client
needs to recognize that these are distinct communications end-points
up through the session layer, even though they connect to the same
filesystem.  So this is why I think of multiple metadata-servers
(in the pNFS sense) as positioned in a spectrum that goes from
loosely-synced replicas up through identical data all the way to
instances that are merely different paths to the same thing.  If
they are a natural and useful stop along the way, I'm inclined to
at least enable them and let people see what they can do.

So one way to enable multiple metadata servers in this framework 
is to have clients not open multiple connections within a single
session (which multiple servers would have trouble dealing with 
due to lack of shared memory to store the last xid for each slot),
but rather multiple sessions for the same clientid.  Multiple 
servers would still have to implement the approrpiate co-ordination
through an underlying cluster file system or otherwise so that
change made at one metadata server are visible at others to
maintain the appearance (or maybe I should say phenomenology to 
avoid committing to a judgement on what the reality really is 
here) of a single fs.

> > So here's the critical question that I think will decide whether
> > this has a chance of flying in the 4.1 timeframe.  When you say you
> > want an attribute that says which server controls this, are you
> > speaking of a hint such that the server will work if the request
> > goes to the "wrong" metadata server?  If so, this might be OK (and
> > it still might be too much). 

> Yes, this is a hint, but I'm not going to ask for it :-)

An attribute that nobody is asking for is probably not going to
make it into v4.1 :-), but in case anyone else is inclined to
ask for attributes like this, let me restate my position.

If you have multiple metadata servers they must be equivalent
so that any specific direction of entities smaller than an fs
must be in the nature of a hint.  Any request should be able to
be directed at any server and work.  Don't assume the converse.
Not every hint would be OK.  If people can't agree on a set of
reasonable hints, then I think it is quite reasonable to avoid
all hints and simply say that when a client has multiple metadata 
servers (i.e. multiple writable replicas that have every coherence 
guarantee specified except that of a single coherent session), it 
should, when using multiple metadata servers for the same fs maintain
locality so that successive requests which are directed at the
same filehandle should generally be sent to the same server
instance.  Anything more specific/detailed could be left to a later 
minor revision.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Jon Haswell <[email-masked]>
Date: Thu, 18 Aug 2005 10:01:48 -0700

Let me strongly endorse Dave's suggestion for a locations_info attribute. I
think it has great value, as Dave says, for multiple cases before one even
gets to pNFS. My particular interest being different servers holding
replicas, probably with no shared hardware resources. I would suggest a few
additional attributes as follows below

Jon Haswell

Dave Novak wrote on 08/18/2005 08:12:10 AM:

<snip>

>     refetch these values and you can assume that the change attribute

How about something along the lines of the following

struct fs_location_info4 {
    utf8str_cis         server<>;
    pathname4           rootpath;
    uint32_t            rank;
    uint32_t            order;
    uint32_t            version;
    uint32_t            mod_time;
    bool                same_fh;
    bool                same_fileid;
    bool                same_change_attrib
};

struct fs_locations_info4 {
    pathname4      fs_root;
    fs_location4   locations<>;
    uint32_t       valid_for;
    uint32_t       must_switch_time;
};

Here would be my definition of the terms

valid_for - this is the number of seconds that the client should consider
this location information to be true for. A well behaved client should
refetch this location info within this time. If on the refetch the client
finds this particular server is nolonger listed as a location then it
should switch to an alternative server within the tiem limited in
must_switch_time.

must_switch_time - the grace time, in seconds, a client is allowed by a
server to switch to a new location when this one becomes invalid. After
this time the client should expect the server to become unavailable,
although ti is not an error for the servr to continue to service the
workload if it can.

The obvious idea behind these two attributes is to give a way for a server
to explicity shed load in an ordered manner, such as when the server needs
to be taken down. The idea being that if clients are directed to switch and
given an ordered amount of time to switch the disruption to the
applications and storm of network traffic if they have to re-establish
context can be minimized.

rank - this value, along with the order attribute given below, are used to
instruct the client on which servers are preferable for it to use for
service. Lower would be better here and the client must select a server
from the list of servers with equal lowest rank unless none of them become
available, in which case it falls back to the list of servers with the next
highest rank.

order - within servers with the same rank this value specifies the server
the client should be directed to, lowest being tried first

Let me give an example of the use of rank and order attributes. Say we have
data on five servers, four used for primary workload, of which we would
prefer the client to use one particular machine, and a fourth only to be
used for emergencies when everythign else is down.

      server            rank  order
      best_server       0     1
      primary_1         1     0
      primary_2         1     1
      primary_3         1     1
      backup_1          2     0

Here the client is specified to pick best_server if that is available. If
it is not available then it is directed to try primary_1 next followed by
either primary_2 or primary_3 and if none of these work it should fall back
to backup_1. It is quite likely that, when dealing with replicas, that the
version of the data on servers within a single rank is identical and the
order is set based on the workload of the particular machine, although the
server is free to use rank and order however it wishes to direct the client
or set the fields to all zero to just let the client choose freely.

version - this would be a numerical value indicating the version of the
data, if this was a replica. If this field has no meaning for this file
system then the server could set it to zero.

mod_time - this would be the modification time for a replica, probably
seconds since epoch or similar. Again if it had not meaning then the server
could default it to current time or zero.

My suggestion would be that the version and mod_time attributes are only
for information purposes and the rank and order attributes should guide the
clients choice of server, unless they are left as all zero by the server in
which case then the client could, at it's discretion, use the version and
mod_time fields to guide it's choice.

same_fh
same_fileid
same_change_attrib

These fields as Dave suggestion to notify the client about the degree of
disruption a switch will cause. One additional point I would add however is
that these bits should be set specific to the server returning the
locations_info fields, i.e. these bits indicate the degree of distruption
when switching from this server to another in the list - it is not
guaranteed that this degree of disruption will occur between a switch
between any random two servers in the list. I think this is a requirement
to handle the case where two primary servers are exporting the same
physical file system but a third is some backup device and while the two
primary file servers may share the same fh and fileids the backup will not,
hence a switch from the primary to a backup will be more disruptive than
between the two primaries.

I would expect the client to use these fields, along with the version and
mod_time values to guide it's choice of server if it is not mandated to go
to a particular server via the rank and order attribute. i.e. a server
handing out a locations_info attribute can choose to absolutely direct the
client on which servers to go to using rank and order or it can choose not
to exercise such close control by leaving rank and order all zero and allow
the client to choose based on version, mod_time, same_fh etc.

> appears performance is OK.

<snip>

Thanks

Jon Haswell
IBM Research
Tel 408 927 2074 Tie 457 2074
Fax 408 927 3030 Tie 457 3030

---

From: Charles Fan <[email-masked]>
Date: Thu, 18 Aug 2005 11:08:56 -0700

Hi,

I would also like to echo Dave's suggestion to consider adding the 
location_info attribute.

Charles

---

From: Marc Eshel <[email-masked]>
Date: Thu, 18 Aug 2005 13:39:06 -0700

I also support the need for locations_info. Brent concern is that it will 
slow down pNFS and I agree with him. Dave, do you think that it should be 
part of the pNFS draft or on a different one. I also want to warn that we 
don't get carried a way with the number of hints that will make a client 
implementation to complicated or will just be ignored.
Marc.

---

From: Brent Welch <[email-masked]>
Date: Thu, 18 Aug 2005 15:13:35 -0700

OK, I follow the argument that this really applies to the existing
replication feature inside NFSv4, which isn't really exploited by
the current generation of servers and clients.  Your set of proposed
attributes in locations_info seems pretty good.  I would add a bit
that indicates if a replica is writable or not (perhaps that already
exists).

I could make this work in my cluster file system, but if I could ask
for the moon I would ask for a per-file redirect capability.  This
reflects my bias, which is that I'd rather have the file state reside
on one server and redirect clients to that server, rather than ping-pong
state among servers as the clients blindly use whatever server they want,
or do server-to-server forwarding.  With pNFS, the client isn't going
to use that server for data transfer, so I think net-net it is better
to keep the server state in one place.  Admittedly, the state ping-pong
or server-to-server forwarding have the benefit of being client transparent.
But, since we are talking about the client-server protocol we could contemplate 
protocol changes that allow the client to help.  Help me, at least.
Again, I don't want to be obstructionist, and casting this as better
support for replication is the right approach.  I'd just like to point
out that it doesn't support all methods of metadata load balancing,
which I guess I have pointed out before.

But, thinking of my cluster of metadata managers as a kind of replication
is a neat view.  Thanks!

[no new material below here]

>>>"Noveck, Dave" said:
 > information is a problem.			=09
 > 	      

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]