Subject: RE: [nfsv4] Locations_info and multiple metadata servers

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 19 Aug 2005 16:28:51 -0400

My plan is to do this as part of a draft covering replication/migration/
referral issues for v4.1.  A lot of that will be clarification and
corrections for existing functionality but there would also be 
a few new suggested attributes of which locations_info is most 
significant.

Even though this would be useful for pnfs, I don't see this as 
basically tied to pnfs or part of pnfs.  There would be a section
that describes how this would integrate with pnfs.  The content
would basically suggest that when pnfs is in use and there are multiple
concurrent v4 servers become concurrent pnfs metadata servers.  If
there is consensus that both pnfs and locations_info belong in the
v4.1 rollup, then there would have to be a bit more text describing
co-ordination between these features.  For example, if there are
multiple metadata servers they must agree on layout contents, etc.
My guess is that this can be done without a separate round of drafts
by getting general consensus on the principles and working together
with Spencer to arrive at the co-ordination text for the rollup 
draft.  Spencer may have other ideas, of course.

As far as possible hints for per-file affinity, I am going to go
ahead without waiting for that discussion.  I'm hoping that the
people with the most interest in that can make some preliminary 
proposals and that we can discuss it on the list and come up with
something that provides the greatest good for the greatest number.
In the draft I'll be working on, I will simply allude to the 
possibility of such hints, unless the discussion comes to a conclusion
much faster than I expect.  So my expectation is that attributes
for such hints would go in a subsequent rollup draft.  My impression
is that that is OK, as long as the attributes are not too complicated
or delayed too long.  I don't want this to delay pNFS so I think we
have to decide that if we can't agree on something reasonable, this
is something that just doesn't happen right now and pNFS goes ahead
without it.

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]]
Sent: Thursday, August 18, 2005 4:39 PM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] Locations_info and multiple metadata servers

I also support the need for locations_info. Brent concern is that it will 
slow down pNFS and I agree with him. Dave, do you think that it should be 
part of the pNFS draft or on a different one. I also want to warn that we 
don't get carried a way with the number of hints that will make a client 
implementation to complicated or will just be ignored.
Marc. 

[email-masked] wrote on 08/18/2005 08:12:10 AM:

> This is the note I've promised about what I was thinking about a 
> possible extension to provide more information about the possible 
> replicas or migration targets and how it that might be useful to
> provide multi-pathing support and somewhat address some of the
> needs of cluster fs implementation in which multiple servers have 
> the capability of acting as metadata servers, for a given fs.
> 
> Brent Welch wrote:
> > I would like to put multiple metadata servers out of scope.
> > What worries me is that there are a number of different
> > models for how to do it, and that features like the attribute
> > you and Marc were discussing don't necessarily apply to all,
> > or that don't go far enough.
> 
> I see your point, but my perspective is a little different.
> I agree that we do not want to take on the task of supporting
> every sort of multi-metadata-server arrangement that people 
> want (or could conceivably want), or even any fixed set of 
> those.  On the other hand, if there are features that make 
> sense for v4.1, that do enable some variants of this (even 
> if not all or most), then I don't see going out of our way 
> to foreclose them, simply because they cross the line into 
> multiple-metadata-server territory. 
> 
> > In particular, for us we'll have multiple NFSv4 servers that
> > export the same file system, and clients can use any of them.
> > But, they have to pick one at mount time so you get fairly
> > coarse load balancing among them.  "It would be nice" if we
> > had a way to disperse clients among the servers, but it
> > isn't strictly necessary for pNFS.  Already, by distributing the
> > data servers we've offloaded the metadata server a great deal.
> > I'm content to stop here, although it would be possible to
> > make things even more efficient with yet more protocol.
> 
> I agree that multiple metadata servers per se are not something that
> we really need to specifically address, but I felt that since we 
> have replication which is a means to provide multiple v4 servers 
> for a filesystem, it follows that we have, in the pnfs context, a
> way to have multiple meta-data servers.  The issue, as it is with 
> the non-pnfs case, is giving the client sufficient information to
> determine which one he should use, and when he should/may switch 
> to a different one, and what the nature of the transition is.  In
> other words, we have a way to specify multiple (typically read-only)
> replicas and giving the client information to make a proper selection
> is a good idea on lots of grounds.  When you have multiple metadata
> servers you are basically in adjoining territory: telling the 
> client which of a set of replicas he might use and how and when.
> 
> RFC 3530 already gives you a set of locations but aside from the 
> migration case, in which the server tells you an fs is gone and 
> is now somewhere else, there is no real guidance about when you
> may or should switch to another server.  You have a list of locations 
> on a number of servers.  What other information do you need?
> 
> Well, my belief is that the client could find a fair amount of 
> additional information pretty helpful, both for initial selection,
> and for switching when appropriate. 
> 
> So I think we could add a locations_info attribute that contains
> what fs_locations does plus some useful additional information.
> The fs_locations attribute would remain for those accustomed to 
> use it, but it be used less and less, even without formally being
> "deprecated".  The locations_info attribute would allow you to 
> determine, for each of the locations.
> 
>     What the is the server's priority for these?.
> 
>     Do two replica represent the exact same data, or they possibly
>     slightly different images (snapshots at different times)?
> 
>     Which of two replicas (that are different) is more up-to-date?
> 
>     Do two replicas have the same fh's (allowing a switch with no
>     disruption due to filehandle volatility)?
> 
>     Do two replicas agree on fileid's (allowing a switch without 
>     breaking applications that use stat and depend on fileid stability
>     and uniqueness)?
> 
>     Are two replicas going to agree on the defintion of the change
>     attribute such that in a transition, you don't have have to
>     refetch these values and you can assume that the change attribute
>     being the same?
> 
> Unless your model is that I will use a replica until it is absolutely
> dead and will then pick a different one (at random) and live with 
> whatever discontinuities that causes (or rather the applications will
> have to live with consequences of the discontinuities) not having this 
> information is a problem. 
> 
> Clients may want to switch when a server is not performing well, as 
> opposed to waiting until death certified by coroner, but to do that 
> it needs information on what kind of switch between replicas is involved
> and RFC3530 lends itself to multiple interpretations about this issue.
> Rather than try to designate a single intepretation as "correct", I 
> think we need to recognize that many of these are valuable in various
> contexts but the client needs to know which one he is dealing with to
> be effective.  If the client has this information he may be able to
> switch replicas to deal with poor performance and switch back if it
> appears performance is OK.
> 
> Carl Burnett suggested an interesting case which I think fits into 
> this basic framework which is simply that you may have two replica
> designations which are simply different paths to the very same 
> realization of a server/fs.  From the viewpoint of the characteristics
> mentioned above, they are two replicas that always have the same data,
> the same inode numbers, the same filehandles, the same scheme for the
> change attribute, and finally the same set of locking data.  Switching 
> between such "replicas" is extremely easy and one might then go to the 
> next step and simply say that no switching is necessary.  Concurrent 
> access to multiple nominal addresses is precisely what the client would
> want to do, when a single path is insufficient.
> 
> When two IP addresses terminate at the same physical server, the client
> may have multiple connections each associated with those distinct
> physical address, all part of the same session.
> 
> So given that basic context, it would seem that the case of a cluster
> fs is one step back from the extreme case (not meant in a pejorative
> sense) of multiple paths to the same fs.  Where you have multiple 
> servers partaking in access to a clustered filesystem, the client
> needs to recognize that these are distinct communications end-points
> up through the session layer, even though they connect to the same
> filesystem.  So this is why I think of multiple metadata-servers
> (in the pNFS sense) as positioned in a spectrum that goes from
> loosely-synced replicas up through identical data all the way to
> instances that are merely different paths to the same thing.  If
> they are a natural and useful stop along the way, I'm inclined to
> at least enable them and let people see what they can do.
> 
> So one way to enable multiple metadata servers in this framework 
> is to have clients not open multiple connections within a single
> session (which multiple servers would have trouble dealing with 
> due to lack of shared memory to store the last xid for each slot),
> but rather multiple sessions for the same clientid.  Multiple 
> servers would still have to implement the approrpiate co-ordination
> through an underlying cluster file system or otherwise so that
> change made at one metadata server are visible at others to
> maintain the appearance (or maybe I should say phenomenology to 
> avoid committing to a judgement on what the reality really is 
> here) of a single fs.
> 
> > > So here's the critical question that I think will decide whether
> > > this has a chance of flying in the 4.1 timeframe.  When you say you
> > > want an attribute that says which server controls this, are you
> > > speaking of a hint such that the server will work if the request
> > > goes to the "wrong" metadata server?  If so, this might be OK (and
> > > it still might be too much). 
> 
> > Yes, this is a hint, but I'm not going to ask for it :-)
> 
> An attribute that nobody is asking for is probably not going to
> make it into v4.1 :-), but in case anyone else is inclined to
> ask for attributes like this, let me restate my position.
> 
> If you have multiple metadata servers they must be equivalent
> so that any specific direction of entities smaller than an fs
> must be in the nature of a hint.  Any request should be able to
> be directed at any server and work.  Don't assume the converse.
> Not every hint would be OK.  If people can't agree on a set of
> reasonable hints, then I think it is quite reasonable to avoid
> all hints and simply say that when a client has multiple metadata 
> servers (i.e. multiple writable replicas that have every coherence 
> guarantee specified except that of a single coherent session), it 
> should, when using multiple metadata servers for the same fs maintain
> locality so that successive requests which are directed at the
> same filehandle should generally be sent to the same server
> instance.  Anything more specific/detailed could be left to a later 
> minor revision.
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4