Subject: RE: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 21 Jun 2005 16:51:07 -0400

> With fs-locations we give the client a list of metadata server 
> that can act as the server for given filesystem and the client 
> can choose which one to use and switch among them at will or 
> because of some failure. 
 
The problem is that in v4.0 as it stands there is an enormous
range of ways that a client can interpret the fs_locations list:
 
"Here is a list of servers and you can switch when there is failure,
after refetching your change attibutes (because change attribute
is within the purview of a particular server)."  (and that is not the
most extreme case of discontinuity -- some people are thinking that 
filehandles and fileids will just change or that you may wind up
with a slighly out-of-data version of the data).
 
"Here is a list of servers and you can switch when there is a 
failure with no discontinuity of access (changes in fh's, stateid's,
fileids, change attributes), although changing is a big deal
and you shouldn't do it without good cause."
 
"Here is a list of servers and you can switch when there is a
failure with no discontinuity of access or even at will since
there is no big cost to switch."
 
"Here is a list of servers and you can access any of these 
servers as you will at the same time (multi-pathing or a 
cluster fs) since they are all effectively the same thing".
 
The problem is that the client only knows about the list and
has no way of knowing which of the statements above is associated
with the list of servers he is getting.  I have been thinking
about a locations_info attribute for 4.1 that would allow the
server to tell the client which of those he meant and also 
give preference information (local vs. remote copies, absolutely
up-to-data vs. slightly out-of-data copies).
 
> is solely within the purview of a particular server
 
> I agree that once you got a layout for a file you should follow 
> the layout instruction and read/write only from the specified 
> nodes and get an error if you don't. 
 
Then it sounds like we are in violent agreement, except maybe for
choice of modal auxiliaries or capitalization.
 
You say "you should follow the layout instruction" and I'm torn
between saying "you SHOULD follow the layout instruction" and
"you MUST follow the layout instruction".
 
You say "[should] get an error if you don't" and I say "the server
SHOULD give you an error if you don't".

	-----Original Message-----
	From: Marc Eshel [mailto:[email-masked]] 
	Sent: Tuesday, June 21, 2005 12:51 PM
	To: Noveck, Dave
	Cc: Dean Hildebrand; Goodson, Garth; [email-masked];
[email-masked]
	Subject: RE: [nfsv4] pNFS issues/changes
(draft-welch-pnfs-ops-02.txt)

	[email-masked] wrote on 06/21/2005 06:39:07 AM:
	
	> Hold on.  I'm not clear exactly how people are supposing that
this
	> will work.
	> 
	> If I have a given server and let us say it is the metadata
server
	> for the root directory of an fs, then with v4 (and so far
pnfs) as
	> it stands it is the metadata server for every every descendent
object
	> until and unless the fsid changes.  If a client were to decide
	> arbitrarily that it could try to do metadata operations on a
different
	> server with that same filehandle, then I would say that that
client 
	> is seriouly confused and we'd like to know about it.
	> 
	> If the server is deciding, in Marc's words, that "nodes in my
cluster 
	> filesystem ... take the roles of metadata server or data
server for 
	> each file independently", how is the client to determine what
the 
	> proper metadata server for each file is?  Are we talking about
some
	> protocol extension beyond what has already been discussed for
pnfs
	> to distribute the data server role?  We haven't talked about
anything
	> to specifically support distribution of the metadata server
role for
	> a filesystem.  I'm not saying that that would be a bad thing
to have, 
	> but it seems to be a new thing and right now I think the pnfs
focus 
	> should be more on nailing down the stuff already discussed.
	
	With fs-locations we give the client a list of metadata server
that can act as the server for given filesystem and the client can
choose which one to use and switch among them at will or because of some
failure. I agree that once you got a layout for a file you should follow
the layout instruction and read/write only from the specified nodes and
get an error if you don't. 
	 
	> It is possible with a cluster filesystem to have multiple
clients
	> each mount different servers as the metadata server and leave
it to
	> the cluster filesystem to provide the coherence for metadata
operations.
	> In that case multiple servers would be acting as metadata
servers for
	> what is really the same filesystem.  However, in that case, I
would
	> argue that we still want the language and the error in Garth's
draft
	> (even if Garth now seems to backing away from it).  Suppose a
client
	> is talking to server A and a file has filehandle X and that
file
	> has two stripes, one with handle P on server B and the other
with
	> handle Q on server C.  Those particular handles (should) give
you 
	> the ability to do data operations and nothing else.  If you do
metadata 
	> operations with them, you should get an error, since you are
doing
	> something that pnfs does not allow.  This is without regard to
the
	> fact that B and C may have the ability to act as metadata
servers
	> for other handles.  However, they should not act as metadata
servers
	> for stripes P and Q since P and Q are stripes and not files.
Even if 
	> you had a file with a single stripe, the handle for that
stripe as a
	> file should be different from the one that only gives the
client the
	> right to do IO, P and P' let's say.  If a client takes a
handle that 
	> gives him the right to do IO (he got it from a layout) and
uses it
	> for metadata operations, he is violating the protocol and
should get
	> an error (and the server SHOULD make sure that he gets one).
	
	Yes, once you started to use a metadata server A for file X you
should stick to it as the metadata server for the file until you are
done with it and follow the layout to use the appropriate data servers.
I am not sure why the restriction on the usage of file handles. It is an
add complication to the server implementation to add information to the
fh that restricts it usage from specific nodes. We should say that the
client should not use fh given by layout for other purposes but not
require an error if it does. 
	
	Marc.     
	
	> 
	> -----Original Message-----
	> From: Goodson, Garth 
	> Sent: Monday, June 20, 2005 6:06 PM
	> To: Dean Hildebrand
	> Cc: [email-masked]
	> Subject: Re: [nfsv4] pNFS issues/changes
(draft-welch-pnfs-ops-02.txt)
	> 
	> 
	> Correct, that is one of the issues I mentioned.  But I think
there are 
	> cases where the data servers are neither stock, nor where it
is 
	> desirable that they service more than I/O requests (i.e.,
metadata 
	> requests) in which case they may return an error (and that
error should 
	> be decided upon).
	> 
	> -Garth
	> 
	> Dean Hildebrand wrote:
	> > If they are stock, how can they return a new error unless
they know
	> > something, in which case they are no longer stock.
	> > Dean
	> > 
	> > On Mon, 20 Jun 2005, Garth Goodson wrote:
	> > 
	> > 
	> >>I agree.  I don't think it need be an error, but if the
system isn't
	> >>designed to handle it, it would be nice if the data servers
did/could
	> >>reply with an error.
	> >>
	> >>
	> >>-Garth
	> >>
	> >>Marc Eshel wrote:
	> >>
	> >>>
	> >>>[email-masked] wrote on 06/20/2005 12:39:59 PM:
	> >>>
	> >>> > 5.1 File Striping and Data Access
	> >>> >
	> >>> > Change: simplify striping layout -- have enum for SPARSE
vs. DENSE
	> >>> > layout instead of skip and start offset
	> >>> >
	> >>> > Issue: think about what error gets returned if a client
performs a
	> >>> > non(READ/WRITE/PUTFH/COMMIT) at a data server; issue:
this may be a
	> >>> > problem if a regular nfsv4 data server is used as it has
no way to
	> >>> > differentiate accesses.
	> >>> >
	> >>>
	> >>>Why is should it be an error in the first place. I would
like all the
	> >>>nodes in my cluster filesystem to take the roles of
metadata server or
	> >>>data server for each file independently.
	> >>>Marc.
	> >>
	> >>_______________________________________________
	> >>nfsv4 mailing list
	> >>[email-masked]
	> >>https://www1.ietf.org/mailman/listinfo/nfsv4
	> >>
	> 
	> _______________________________________________
	> nfsv4 mailing list
	> [email-masked]
	> https://www1.ietf.org/mailman/listinfo/nfsv4
	> 
	> _______________________________________________
	> nfsv4 mailing list
	> [email-masked]
	> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Marc Eshel <[email-masked]>
Date: Tue, 21 Jun 2005 15:27:20 -0700

> SHOULD give you an error if you don't".

I say should follow and not MUST follow because I am trying to avoid the 
complication to the server if it MUST enforce this rule which might not be 
a problem for the server in the first place. For example, I might want to 
allow the read of the same large file(no caching) from one set of data 
server for client A and a different set of data server for client B. Now 
to enforce the above rule the server need to some how encode into the file 
handle information about which client can read what from which data 
server? I prefer not have this added extra work on the server and just say 
that the client should follow the rule to guaranty successful operation. 
Marc.

---

From: Garth Goodson <[email-masked]>
Date: Tue, 21 Jun 2005 15:33:30 -0700

> that the client should follow the rule to guaranty successful operation. 

I think your example can be handled by giving client A and client B 
different layouts for the same file.  The filehandles in the different 
layouts can be different as would be the device IDs.  As long as the 
metadata server controls the sharing modes (read-only vs. read/write) 
there shouldn't be a problem.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Tue, 21 Jun 2005 15:45:51 -0700

> there shouldn't be a problem.

I know I can do it. I just don't want to make sure (enforce the rule) that 
each client is using the file handles to read only from the specified data 
server.
Marc.

Marc.

---

From: Garth Goodson <[email-masked]>
Date: Tue, 21 Jun 2005 16:54:01 -0700

Marc Eshel wrote:
>>>"Noveck, Dave" <[email-masked]> wrote on 06/21/2005 01:51:07 
> 
> PM:
> 
>>>
>>>I say should follow and not MUST follow because I am trying to avoid 
> 
> the 
> 
>>>complication to the server if it MUST enforce this rule which might 
> 
> not be 
> 
>>>a problem for the server in the first place. For example, I might want 
> 
> to 
> 
>>>server for client A and a different set of data server for client B. 
> 
> Now 
> 
>>>to enforce the above rule the server need to some how encode into the 
> 
> file 
> 
>>>server? I prefer not have this added extra work on the server and just 
> 
> say 
> 
>>>that the client should follow the rule to guaranty successful 
> 
> operation. 
> 
>>>Marc.
>>

Ok, that is a valid concern (not having to propagate layouts to the data 
servers to validate that I/Os are coming from the correct clients).  I 
guess the object guys get around this by encoding the layout/device IDs 
into the capability that is handed back to the client with the layout.

It has been marked as an open issue...

-Garth

---

From: Benny Halevy <[email-masked]>
Date: Sun, 26 Jun 2005 21:14:26 +0300

On 6/22/2005, Garth Goodson wrote:
[snip]
>>
>> I know I can do it. I just don't want to make sure (enforce the rule) 
>> that each client is using the file handles to read only from the 
>> specified data server.
>> Marc.
>>
>> Marc. 
> 
> 

T10's object capabilities model does not encode the client 
identity, nor the layout/device IDs into the capability so 
the object storage device (OSD) have no way to verify that 
an I/O request came from the "correct" client (e.g. the 
client that got the cap could have given it to another 
client and it would just work), yet the capability is 
generated for each device, so a cap for one device wouldn't 
work on another device if the two devices have different 
device keys.