Subject: RE: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 21 Jun 2005 09:39:07 -0400

Hold on.  I'm not clear exactly how people are supposing that this
will work.

If I have a given server and let us say it is the metadata server
for the root directory of an fs, then with v4 (and so far pnfs) as
it stands it is the metadata server for every every descendent object
until and unless the fsid changes.  If a client were to decide
arbitrarily that it could try to do metadata operations on a different
server with that same filehandle, then I would say that that client 
is seriouly confused and we'd like to know about it.

If the server is deciding, in Marc's words, that "nodes in my cluster 
filesystem ... take the roles of metadata server or data server for 
each file independently", how is the client to determine what the 
proper metadata server for each file is?  Are we talking about some
protocol extension beyond what has already been discussed for pnfs
to distribute the data server role?  We haven't talked about anything
to specifically support distribution of the metadata server role for
a filesystem.  I'm not saying that that would be a bad thing to have, 
but it seems to be a new thing and right now I think the pnfs focus 
should be more on nailing down the stuff already discussed.

It is possible with a cluster filesystem to have multiple clients
each mount different servers as the metadata server and leave it to
the cluster filesystem to provide the coherence for metadata operations.
In that case multiple servers would be acting as metadata servers for
what is really the same filesystem.  However, in that case, I would
argue that we still want the language and the error in Garth's draft
(even if Garth now seems to backing away from it).  Suppose a client
is talking to server A and a file has filehandle X and that file
has two stripes, one with handle P on server B and the other with
handle Q on server C.  Those particular handles (should) give you 
the ability to do data operations and nothing else.  If you do metadata 
operations with them, you should get an error, since you are doing
something that pnfs does not allow.  This is without regard to the
fact that B and C may have the ability to act as metadata servers
for other handles.  However, they should not act as metadata servers
for stripes P and Q since P and Q are stripes and not files.  Even if 
you had a file with a single stripe, the handle for that stripe as a
file should be different from the one that only gives the client the
right to do IO, P and P' let's say.  If a client takes a handle that 
gives him the right to do IO (he got it from a layout) and uses it
for metadata operations, he is violating the protocol and should get
an error (and the server SHOULD make sure that he gets one).

-----Original Message-----
From: Goodson, Garth 
Sent: Monday, June 20, 2005 6:06 PM
To: Dean Hildebrand
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

Correct, that is one of the issues I mentioned.  But I think there are 
cases where the data servers are neither stock, nor where it is 
desirable that they service more than I/O requests (i.e., metadata 
requests) in which case they may return an error (and that error should 
be decided upon).

-Garth

Dean Hildebrand wrote:
> If they are stock, how can they return a new error unless they know
> something, in which case they are no longer stock.
> Dean
> 
> On Mon, 20 Jun 2005, Garth Goodson wrote:
> 
> 
>>I agree.  I don't think it need be an error, but if the system isn't
>>designed to handle it, it would be nice if the data servers did/could
>>reply with an error.
>>
>>
>>-Garth
>>
>>Marc Eshel wrote:
>>
>>>
>>>[email-masked] wrote on 06/20/2005 12:39:59 PM:
>>>
>>> > 5.1 File Striping and Data Access
>>> >
>>> > Change: simplify striping layout -- have enum for SPARSE vs. DENSE
>>> > layout instead of skip and start offset
>>> >
>>> > Issue: think about what error gets returned if a client performs a
>>> > non(READ/WRITE/PUTFH/COMMIT) at a data server; issue: this may be a
>>> > problem if a regular nfsv4 data server is used as it has no way to
>>> > differentiate accesses.
>>> >
>>>
>>>Why is should it be an error in the first place. I would like all the
>>>nodes in my cluster filesystem to take the roles of metadata server or
>>>data server for each file independently.
>>>Marc.
>>
>>_______________________________________________
>>nfsv4 mailing list
>>[email-masked]
>>https://www1.ietf.org/mailman/listinfo/nfsv4
>>

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Marc Eshel <[email-masked]>
Date: Tue, 21 Jun 2005 09:50:44 -0700

> should be more on nailing down the stuff already discussed.

With fs-locations we give the client a list of metadata server that can 
act as the server for given filesystem and the client can choose which one 
to use and switch among them at will or because of some failure. I agree 
that once you got a layout for a file you should follow the layout 
instruction and read/write only from the specified nodes and get an error 
if you don't.
 
> an error (and the server SHOULD make sure that he gets one).

Yes, once you started to use a metadata server A for file X you should 
stick to it as the metadata server for the file until you are done with it 
and follow the layout to use the appropriate data servers. I am not sure 
why the restriction on the usage of file handles. It is an add 
complication to the server implementation to add information to the fh 
that restricts it usage from specific nodes. We should say that the client 
should not use fh given by layout for other purposes but not require an 
error if it does.

Marc.

---

From: Garth Goodson <[email-masked]>
Date: Tue, 21 Jun 2005 09:59:23 -0700

> stick to it as the metadata server for the file until you are done with 
> it and follow the layout to use the appropriate data servers. I am not 
> sure why the restriction on the usage of file handles. It is an add 
> that restricts it usage from specific nodes. We should say that the 
> client should not use fh given by layout for other purposes but not 
> require an error if it does.
> 
> Marc.    
> 

Currently, I'm not sure that we can require an error (as much as I would 
like to require it).  Some people want to be able to use unmodified 
NFSv4 servers as the data servers.  If this is allowed those data 
servers will not know to return an error.  This needs to be worked out. 
  I still believe that it would help the client if an error could be 
returned (as Dave pointed out), especially if the data server can not 
service the metadata operation.

-Garth

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 21 Jun 2005 13:12:26 -0400

At 12:59 PM 6/21/2005, Garth Goodson wrote:
>servers will not know to return an error.

In the presence of a session (which is guaranteed by pNFS, right?), the
v4 server can easily remember whether the client previously requested
and received a device list. Of course, this doesn't prove that a given
client request is non-conflicting (the client could be mixing regular v4
and pNFS traffic on the session), but with a little common sense
specsmanship I think it could be forbidden on a per-pNFS-session
basis.

In fact, I think it would be a good idea, since the server can and should
be able to tune its resources to match the fact that clients won't be
doing much data transfer to the metadata server.

Tom.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 21 Jun 2005 10:13:51 -0700

> service the metadata operation.
If the data server can not service the metadata operation you will get an 
error. The only problem is if it can but it should not. I don't see why we 
would like to restrict it.
Marc.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 21 Jun 2005 11:11:38 -0700

> doing much data transfer to the metadata server.
Yes it would be nice to configure some nodes to be just data server nodes 
but it would also be nice and useful to distribute all metadata and data 
server among all the nodes. The server has the control using exportfs and 
fs-locations to export its metadata nodes and layout information to list 
the data servers. If we say that the file handles given by layout should 
only be used for READ/WRITE/PUTFH/COMMIT it should be enough. I don't see 
the need for additional restrictions. If you do please explain why.
Marc.