Subject: Re: [nfsv4] pNFS, OPEN principals and ACLs

From: Nicolas Williams <[email-masked]>
Date: Wed, 21 Sep 2005 14:17:37 -0500

On Wed, Aug 31, 2005 at 12:08:45PM -0700, Mike Eisler wrote:
> --- Nicolas Williams <[email-masked]> wrote:
> > On Tue, Aug 30, 2005 at 08:22:39PM -0700, Mike Eisler wrote:
> > > What of the issue Garth raised? I propose that we introduce
> > > two new operations, GETFD and PUTFD. Like GETFH, GETFD is invoked in the
> > > same COMPOUND as OPEN, after OPEN. GETFD returns a file descriptor.
> > > PUTFD injects a file descriptor. Unlike file handles, file descriptors
> > > do not persist. One a file is CLOSEd, the file descriptor used
> > > in the CLOSE is disposed of. PUTFD is used for any operation on
> > > a regular file that takes a stateid. PUTFD is used instead of PUTFH.
> > ...
> > 
> > Sounds close to an access token; why not deal with NFS as the backend
> 
> It is solving the same problem as the access token, but it isn't an access token.

So, I don't know which is better, but either way you'll need some
meta-data server<->backend server coordination, either a protocol for
informing backend servers of new and closed/revoked FDs or a some key
distribution (possibly done manually).

I don't care which it is so much as that interop be feasible, so if a
backend protocol is required then IMO it ought to be specified here as
well.

Nico
-- 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Wed, 21 Sep 2005 13:25:53 -0700 (PDT)

> distribution (possibly done manually).

I don't think so.

When the pNFS client issues an OPEN to the the data server, the data server
checks with the metadata server (assuming that's required by the
server implementation) whether the principal issuing the OPEN is permitted
the access. If affirmative, the data server allows the GETFD. The
The client can then use the FD on subsequent I/O without the data server
consulting with the meta dataserver. Even if the ACL on the file changes,
it shouldn't make any difference. The situation is analagous to a
UNIX process opening a file, another process changing the ACL on the
filer, and the first process being allowed to access the file via its
open descriptor, forever, until the process closes the descriptor.
 
> backend protocol is required then IMO it ought to be specified here as

Various implementors are proceeding with their own parallel file systems.
They don't have to, and probably don't want to use the NFSv4 working
group's prescribed metadata server and dataserver protocol.

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Wed, 21 Sep 2005 17:43:41 -0400

> open descriptor, forever, until the process closes the descriptor.

the pNFS client is not allowed to issue OPEN/CLOSE to the data server, so i 
don't understand what you mean. perhaps you mean that the pNFS client presents 
a PUTFD to the data server in the compound with the IO request. the first time 
the data server sees a PUTFD it checks with the metadata server to see if the 
principal issuing the IO is permitted the access.

upon close, the metadata server will need to inform the data servers that the 
FD is no longer valid.

if we change the spec to allow the pNFS client to issue OPEN/CLOSE to the data 
server (which i think is a bad idea), then the pNFS client has a stateid to 
present to the data server for IO, using the existing NFSv4.0 checks.

-->Andy

---

From: Mike Eisler <[email-masked]>
Date: Wed, 21 Sep 2005 15:29:28 -0700 (PDT)

> the pNFS client is not allowed to issue OPEN/CLOSE to the data server, so i 

Garth wrote on August 29:

"For 4.1 and as a dependency for pNFS, I would like to propose that the 
principal used for OPEN is the same principal as used for I/O.

This has been brought up a number of times and as currently worded 
within the v4 spec (no requirement that READs and WRITEs issued for an 
open file have the same credentials as the OPEN itself) poses undue 
overhead on the data servers to access check each I/O based on the 
principal within the operation against all principals within the ACL. 
It also means that ACL changes must be propagated synchronously to each 
data server."

If data server's don't accept OPENs, then how can they interpret stateids
per Garth's proposal? If they don't accept OPENs then why, in abscence of
Garth's proposal or my counter proposal, do ACL changes need to be
propagated synchronously to the data server? Why do data servers have to
check access on each I/O?

> principal issuing the IO is permitted the access.

I may be confused, but the security model for pNFS appears to be even more 
confusing.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 21 Sep 2005 17:46:34 -0500

> open descriptor, forever, until the process closes the descriptor.

Er, does this imply that FHs have to match on both meta-data and backend
servers?

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 21 Sep 2005 17:59:44 -0500

> Er, does this imply that FHs have to match on both meta-data and backend

Er, yes, I'm stupid.  I'll go back to lurking now.

---

From: Garth Goodson <[email-masked]>
Date: Wed, 21 Sep 2005 16:21:46 -0700

> check access on each I/O?

To maintain security, data servers need to, in some way, validate the 
principle has access to perform the I/O.  As stands, in order for data 
servers to do this, they need to have an ACL (or some permissions) to 
check against.  In order for them to have this permission state, it must 
be propogated from the metadata server.  In order to more closely mimic 
regular V4, stateids obtained from performing the OPENs at the metadata 
server are to be used on the data servers.  This requires a backend 
protocol to transfer this information.

To summarize, clients do an OPEN at the metadata server, the metadata 
server propogates information regarding that OPEN to the data servers. 
If the OPEN principle is the same as the principle used for I/O, the 
data server need only check that the principle matches the stateid (or 
some cookie encoded into the FH, or some FD that is passed around, or 
some capability) and that the stateid is valid, if not, the data server 
also needs to validate the principle against those found in the ACL (the 
access check per I/O).

Your FD proposal can eliminate the ACL check on the data server, however 
it requires the FD to be propogated from the metadata to the data 
server.  I believe that this is similar to tying the principle to a 
stateid, however this approach has problems with delegations (as was 
mentioned in a previous msg).

However, I also believe, that we could add an operation that took a 
principle, a delegation stateid, and a FH, did the access check, and 
returned a different stateid that was tied to the principle, thereby 
producing a FD of sorts that was tied to a principle and permitted access.

-Garth

---

From: David Robinson <[email-masked]>
Date: Wed, 21 Sep 2005 18:30:11 -0500

> present to the data server for IO, using the existing NFSv4.0 checks.

Why is it a bad idea? In the current model the OPEN to the metadata
server effectively does a 'proxy' open on the data server and
returns back the information needed to talk to the data server
without an explicit OPEN.

You can make efficency arguments that it saves a round trip,
although in an I/O intensive workload it will be lost in the noise.
In either case there is clearly some back end coordination
needed. But by doing an explicit OPEN to the data server it
simplifies the principal and stateid issues.  What am I missing?

	-David

---

From: Garth Goodson <[email-masked]>
Date: Wed, 21 Sep 2005 17:31:48 -0700

> without an explicit OPEN.

This is an artifact of the implementation.  If you are building pNFS 
above some type of clustered filesystem, multiple OPENs are in fact not 
needed, and would be more harmful than good.  For these systems, the 
underlying FS takes care of distributing locks, metadata (including 
stateids), ACLs, etc and forcing an alternative approach on top of this 
would be harmful.

There are many ways in which this information can be propogated, forcing 
an OPEN to every data server, limits the options.  As well, these 
cluster interconnects can be much more efficient than relying on the 
client for propagation.  Lastly, I could imagine it may simplify 
recovery from failed client OPENs.

-Garth

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 21 Sep 2005 21:06:53 -0400

> recovery from failed client OPENs.

What kind of OPENs would these be anyway? The layouts only return
filehandle information, not pathnames, on the data servers.

Would these OPENs carry share lock information? If so, how would the
metadata server and the data servers ensure consistency?

Cheers,
  Trond

---

From: David Robinson <[email-masked]>
Date: Wed, 21 Sep 2005 23:52:31 -0500

> filehandle information, not pathnames, on the data servers.

Clearly the current layout information would not be useful, but
a new layout format that returned servera:/patha,serverb:/pathb,...
could be returned to the client, from which the OPEN would return
the filehandle and stateid as it normally does.

> metadata server and the data servers ensure consistency?

The metadata server and data servers must cooperate and share
information about which data servers can server which portions
of a file. It can't allow two data servers to control the same
portion or bad contention issues would exist.  There still needs to
be an evaluation of the OPEN on the metadata server and
the metadata server still needs to communicate to the data servers
that a particular client has access. The OPENs to each data server
should have the same openowner and share lock information
which needs to be validated if it has access.  This is no different than
the validation of the filehandle that we have now. But what it does
allow that each data server has its own stateid, credential validation,
and recovery without involving the metadata server.

	-David

---

From: Marc Eshel <[email-masked]>
Date: Wed, 21 Sep 2005 21:56:03 -0700

Garth Goodson wrote on 09/21/2005 05:31:48 PM:
> >> if we change the spec to allow the pNFS client to issue OPEN/CLOSE to 

> >> the data

I agree with most of this statement with the exception of "(including 
stateid}". I will describe the Linux implementation but this is also true 
for other systems. The physical file system can see only what is defined 
in the VFS interface and stateids are NOT part of it. So yes the cluster 
file system can support the distributed locking and permission checks and 
it can communicate with the other nodes in the cluster but it knows 
nothing about stateids. The NFS server can export different file systems 
so although it would be nice to use the fs cluster connectivity only the 
NFS metadata server can communicate stateids to the NFS data server. but I 
don't think that it should.

> recovery from failed client OPENs.

I think that the data should propagate through the client the fact that 
the cluster is interconnected doesn't mean that it is free. pNFS is a 
performance item and the addition of messages by the back-end for every 
open close is taking a way from it. All the information should be 
propagated using some new pOpen (using FH and not pathname), or PUTFD or 
just use special stateids of 0's and 1's when ever possible(for example 
read only fs). This information can be provided to the data server on the 
first read by the client using a compound RPC and avoid the extra 
messages. 
Marc.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 21 Sep 2005 22:31:21 -0700 (PDT)

> check against.  In order for them to have this permission state, it must 
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> be propogated from the metadata server.  In order to more closely mimic 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Right. That property would seem to be common to all parallel 
and/or clustered file system that back pNFS. 

> protocol to transfer this information.

But not just transfer. Such a protocol needs to inform data servers
when the stateids disappear on the metadata server, due to say
metadata server reboot.

> mentioned in a previous msg).

I was thinking that the data servers would field the OPEN, and thus
generate the FD on their own.

I can see arguments why that might be less than optimal.

How is this:

Clients issue OPEN to the metadata server and get back a stateid.
The clients obtain the FH for the same file from the data server.
The clients then do:

    PUTFH, PUTSTATEID, ACCESS, GETFD

The FD maps to a tuple 
 { FH, access rights [ one of: read, write, read/write, none ], principal }

If the backend file system is clustered with distributed locking etc, then
GETFD doesn't need to leave the data server; a token or delegation-like
mechanism in the clustered file system ensures the information is
available to the data server. 

If the backend file system is
a simpler parallel file system (one which tries to avoid 
all that distributed lock state), then it can determine the
access rights via a backend protocol call from
the data server to the metadata server which when given the principal,
the open stateid and the file handle, determines what rights the principal
has (which are determined by the share reservation
type and the ACL). The FD can even be generated 
on the metadata server for simplicity
of implementation, but that's a pNFS implementation choice. The
reboot of the metadata server doesn't require it to contact
the data server to invalidate FDs or stateids. There's no reason to
invalidate the FD at all as far as I can tell.

The I/O can use the special stateids, because all the
necessary info is conveyed in the FD.

> producing a FD of sorts that was tied to a principle and permitted access.

I'm not sure I understand why this would be a delegation stateid, since
not all OPENs on the metadata server will generate a delegation. But
otherwise your belief matches my revised GETFD idea.

Maybe the FD concept is unnecessary and this 
can all be handled via a stateid.
But Carl B at least seems resistant to the idea of tying
stateids to principals.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 22 Sep 2005 09:20:09 -0400

> the filehandle and stateid as it normally does.

That means that the data servers are suddenly required to expose path
information to the clients. That again means that you suddenly are
vulnerable to rename operations on the data server. Perhaps not a
problem in the case where data servers are "intelligent disks" with no
local users, but definitely a problem in the cluster filesystem case.

It also adds a lot of extra overhead in the form of a full path walk on
the data server.

> and recovery without involving the metadata server.

Sorry? I don't see how this can be the case at all.

If credential validation isn't done by the metadata server, then you
will also have to add some fairly complex backend stuff to ensure that
file mode information never gets out of sync between the metadata server
and the data servers. In the case of clusterfses then someone else has
already done that work for you, (but they don't need the special OPEN
anyway) however in generic case...

It also adds extra complexity due to the fact that you are
decentralising your state model. For instance, the success of the
operation open("foo",O_CREAT|O_WRONLY, S_IRUSR) depends on knowing that
your process was the one that created the file. How does a client
communicate this "special open privilege" to the data servers?

Then there is the overhead of having to do the full
setclientid/renew/... dance with a new set of servers. Plus the overhead
of a full new set of open owners.

....

Cheers,
  Trond

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Thu, 22 Sep 2005 10:31:19 -0400

> be propogated from the metadata server.

this is fine. ACLs don't change that often.

> In order to more closely mimic 

the stateid check is painful. bruce fields emailed the list about this issue, 
here is what he said;

[email-masked] said:
> Why shouldn't pNFS clients use only special stateid's for all IO?
> >From http://www.ietf.org/internet-drafts/draft-welch-pnfs-ops-03.txt:
> 	"The client MUST use the stateid representing open or lock state
> 	as returned by an earlier metadata operation (e.g., OPEN, LOCK),
> 	or a special stateid to perform I/O on the data-servers (as in
> 	regular NFSv4)."
> I don't understand why we're requiring this.
> Stateid's perform two purposes that I can think of. First, they provide
> protection against replays that cross state-transition boundaries.  (For
> example, if a client flushes some write data and then sends an unlock, there
> is a risk those writes could be replayed after the unlock.  The stateid
> change on unlock allows the server to reject those replays.)

> Second, stateid's tie I/O to lockowners, allowing the server to determine
> whether a given I/O request conflicts with an existing mandatory file lock.

> I believe sessions solve the replay problem completely.

> The second problem should also be solveable using sessions: the session
> allows the server to determine the client that sent a given I/O request, thus
> allowing it to recognize inter-client conflicts.  If the client itself can be
> relied on to handle conflicts between local processes, then we're done.

> What am I missing?

> The current draft's stateid requirement necessitates a high-performance
> nfsv4-specific server-to-server protocol, which seems likely to complicate
> pNFS implementation.  So it would obviously be nice if we could eliminate
> that requirement.

> At the very least I think the draft needs a clear statement of the problem
> that stateid's are meant to solve.  Maybe I'm missing it....

> --b. 

so, does the combination of sessions and client co-operation allow pNFS to 
avoid the pushing of stateids to data servers?