Subject: RE: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 3 Aug 2005 17:45:05 -0400

Although I can see that one might want that, the question that this
raises is whether that kind of addition (and probably ones that follow
in its wake) are such as to confirm some peoples original feeling,
that multiple metadata servers best considered out of scope for 
pNFS in v4.1.

I was hoping that there could be some synergy with some very small
stuff that is basically in the migration/replication space.  There has  
been some interest in multiple server locations and how those might 
be assimilated to multiple pathing (i.e. telling the server when 
multiple server locations are "really" the same).  It looked to me
like a cluster file system with multiple metadata servers might be
assimilated into such a spectrum since there are multiple and they
are "really" the same, for some definition of "really".  So that makes
any reference to the fact that they really are different somewhat
problematic.

So here's the critical question that I think will decide whether 
this has a chance of flying in the 4.1 timeframe.  When you say you
want an attribute that says which server controls this, are you 
speaking of a hint such that the server will work if the request 
goes to the "wrong" metadata server?  If so, this might be OK (and 
it still might be too much).  

The reason I lay so much stress on this issue is that when it is a 
hint there are two sorts of problems that get a lot simpler in this
case.  The first is that issues of when this can change no longer are
critical.  If you have a value which becomes wrong, no big deal.  The 
second is that when you have a request with multiple filehandles, the
issue of what to do when they are controlled by different servers
is no longer a killer.  If they are different, send it to one or the
other or even some third server.  If you have a cluster filesystem
any request received by a wrong server can be dealt with even if it
is not the case that there is no perfectly right server in all respects.

-----Original Message-----
From: Brent Welch [mailto:[email-masked]] 
Sent: Tuesday, August 02, 2005 8:16 AM
To: Noveck, Dave
Cc: Marc Eshel; Goodson, Garth; [email-masked]
Subject: Re: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt) 

 > What we are talking about here is an attribute but is one that has
 > significant implications so it certainly isn't tiny, so if we are
going
 > to add this, "'twere well it were done quickly" or at least
realtively
 > so.

I feel we need more than just an attribute that lists the
metadata servers for a file system.  You'll also want, or at
least I do, some way to say "this file is controlled by
that server" - in other words, a more specific thing than
"this file system is controlled by that group of servers"

[nothing new below here]

>>>"Noveck, Dave" said:
 > 
 > Mark Eshel wrote:
 > > This sound like a good idea. Do you still have time to start new
 > > extensions for 4.1?
 > 
 > I'm not sure whether you mean to fit within some deadline for 4.1
 > extensions (is it already too late?) or time for me to actually do
 > the work?
 > 
 > If the first, the answer is "I don't know".
 > 
 > And if the second, the answer is "I don't know" as well.
 > 
 > Let me try to clarify these. :-)
 > 
 > The issue of when to close the door on new material is one the that
 > the working group has to decide and presumably will be discussed in
 > Paris.  I've discussed this with Spencer who feels there will still
 > be room for "small" items for a while, with the definition of "small"
 > being somewhat elusive.  My feeling is that we do have to close the =
 > door
 > at some point but the nature of what we are doing does not really
 > admit of a fixed cutoff.  I do think that new big items should be
 > off the table but that for smaller items such as new attributes, we
 > have room for a while (with the definition a "a while" being
something
 > the working group needs to decide on).  Also, the decision on any
 > particular item is up to the working group and it is just a fact
 > that things that we think are important and needed will wind up
 > looking smaller and more manageable to us than things that just seem
 > like a nice idea.  As time goes on, we will have to raise the bar
 > though or else v4.1 will never get done.  Expect to hear me (and
 > everybody) on all sides of this issue as each case will pose its
 > own unique issues.
 > 
 > What we are talking about here is an attribute but is one that has
 > significant implications so it certainly isn't tiny, so if we are
going
 > to add this, "'twere well it were done quickly" or at least
realtively
 > so.
 > 
 > As to my time, as I indicated, I'm unsure about this.  I've just come
 > back from vacation and my available free time is even more
non-existent
 > than usual.
 > 
 > Trying to come up with a schedule for an initial I-D just doesn't
seem
 > possible.  What I will try to do over the next week is to come up
with
 > some mail laying out a possible approach, so that the working group
can =
 > 
 > discuss it.  If there is sufficient interest, I will try to square
the
 > circle as far as my own time commitment to get an I-D out is
concerned.
 > 
 > 
 > -----Original Message-----
 > From: Marc Eshel [mailto:[email-masked]]
 > Sent: Monday, July 18, 2005 10:14 PM
 > To: Noveck, Dave
 > Cc: Dean Hildebrand; Goodson, Garth; [email-masked];
 > [email-masked]
 > Subject: RE: [nfsv4] pNFS issues/changes
(draft-welch-pnfs-ops-02.txt)
 > 
 > 
 > "Noveck, Dave" <[email-masked]> wrote on 06/21/2005 01:51:07
=
 > PM:
 > 
 > > > With fs-locations we give the client a list of metadata server
 > > > that can act as the server for given filesystem and the client
 > > > can choose which one to use and switch among them at will or
 > > > because of some failure.
 > >
 > > The problem is that in v4.0 as it stands there is an enormous
 > > range of ways that a client can interpret the fs_locations list:
 > >
 > > "Here is a list of servers and you can switch when there is
failure,
 > > after refetching your change attibutes (because change attribute
 > > is within the purview of a particular server)."  (and that is not
the
 > > most extreme case of discontinuity -- some people are thinking that
 > > filehandles and fileids will just change or that you may wind up
 > > with a slighly out-of-data version of the data).
 > >
 > > "Here is a list of servers and you can switch when there is a
 > > failure with no discontinuity of access (changes in fh's,
stateid's,
 > > fileids, change attributes), although changing is a big deal
 > > and you shouldn't do it without good cause."
 > >
 > > "Here is a list of servers and you can switch when there is a
 > > failure with no discontinuity of access or even at will since
 > > there is no big cost to switch."
 > >
 > > "Here is a list of servers and you can access any of these
 > > servers as you will at the same time (multi-pathing or a
 > > cluster fs) since they are all effectively the same thing".
 > >
 > > The problem is that the client only knows about the list and
 > > has no way of knowing which of the statements above is associated
 > > with the list of servers he is getting.  I have been thinking
 > > about a locations_info attribute for 4.1 that would allow the
 > > server to tell the client which of those he meant and also
 > > give preference information (local vs. remote copies, absolutely
 > > up-to-data vs. slightly out-of-data copies).
 > >
 > This sound like a good idea. Do you still have time to start new
 > extensions for 4.1?
 > Marc.
 > 
 > _______________________________________________
 > nfsv4 mailing list
 > [email-masked]
 > https://www1.ietf.org/mailman/listinfo/nfsv4
 > 
 > 

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Brent Welch <[email-masked]>
Date: Thu, 04 Aug 2005 04:05:59 -0700

> that multiple metadata servers best considered out of scope for

I would like to put multiple metadata servers out of scope.
What worries me is that there are a number of different
models for how to do it, and that features like the attribute
you and Marc were discussing don't necessarily apply to all,
or that don't go far enough.

In particular, for us we'll have multiple NFSv4 servers that
export the same file system, and clients can use any of them.
But, they have to pick one at mount time so you get fairly
coarse load balancing among them.  "It would be nice" if we
had a way to disperse clients among the servers, but it
isn't strictly necessary for pNFS.  Already, by distributing the
data servers we've offloaded the metadata server a great deal.
I'm content to stop here, although it would be possible to
make things even more efficient with yet more protocol.

 > it still might be too much). 

Yes, this is a hint, but I'm not going to ask for it :-)
[nothing new below here]

 > is not the case that there is no perfectly right server in all  > respects.

 >  > be room for "small" items for a while, with the definition of  > "small"
 >  > being somewhat elusive.  My feeling is that we do have to close the  > =
 >  > door
 >  > As to my time, as I indicated, I'm unsure about this.  I've just  > come
 >  > > most extreme case of discontinuity -- some people are thinking  > that

www.panasas.com
[email-masked]