Subject: Re: [nfsv4] pNFS, OPEN principals and ACLs

From: [email-masked]
Date: Mon, 29 Aug 2005 16:20:30 -0400 (EDT)

Ok, so the client should use the same principal credentials as was used
for the Open that returned the stateid in the I/O Op.

I can think of a couple of things:
1 - At one point, byte range Lock Op stateids were in the list for use
    in I/O Ops. I assume this would no longer be recommended practice?

2 - I would assume that the Delegation stateid returned by the Open would
    be fine. Suppose the client no longer has the principal credentials
    for the principal that did the Open? Since a Delegation can persist
    on a client indefinitely, the principal that did the Open could have
    long since logged out, gone on vacation, etc.

rick

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 29 Aug 2005 16:59:34 -0400

>     in I/O Ops. I assume this would no longer be recommended practice?

Why not?

>     long since logged out, gone on vacation, etc.

This could also happen when the client doesn't have any other principal
with access to the file to fall back on.

I think the best the client can do is try to return the delegation
before losing the credential.

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 29 Aug 2005 14:31:19 -0700

>     in I/O Ops. I assume this would no longer be recommended practice?

??? Do you mean delegation stateids?

>     long since logged out, gone on vacation, etc.

Delegations can (and will) be forcibly revoked if a client doesn't want
to give it back, so that is not really an issue here.

The big issue is rather whether or not a delegation should be tied to a
single credential. It is certain that all current clients will assume
that delegations are tied to machines, and not credentials.

Note that if the objective is just to ensure that pNFS doesn't have to
check authorisation on every I/O to the disks (at least on a files based
backend) then the delegation will allow the back end to cache ACCESS
information just like the client can.
IOW: the data server can just check once with the metadata server, and
then cache that information.

Cheers,
  Trond

---

From: Garth Goodson <[email-masked]>
Date: Mon, 29 Aug 2005 14:34:53 -0700

>>    in I/O Ops. I assume this would no longer be recommended practice?
This would just require the principal to match the stateid (as in the 
OPEN case).

> before losing the credential.
This is definitely trickier, and what Bruce suggests is an option.

I'm not as familiar with current implementations, but I/O done under a 
delegation seems like it is being done on the client's behalf (as 
opposed to a particular user) and that client is validating the I/O to 
its user/app.  Although, a regular v4 server would be able to validate
the principal on each I/O, this wouldn't be able to occur using this 
approach in pNFS.  It may be good enough to either used cached 
credentials (as Trond suggests), or for the data server to be able to 
tie the stateid back to client through the RPCSEC_GSS session.

-Garth

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 29 Aug 2005 14:43:38 -0700

må den 29.08.2005 Klokka 14:34 (-0700) skreiv Garth Goodson:
> It may be good enough to either used cached 

Note that the delegation is not a promise by the server that just anyone
on that machine can access the file. It is a promise that you can cache
non-conflicting OPENs because no-one is able to change the current ACLs
without causing the delegation to be revoked.

IOW: tying the stateid back to the client is not by itself sufficient to
guarantee correct access checking.

Cheers,
  Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 29 Aug 2005 18:13:06 -0400

> guarantee correct access checking.

Plus there's no way to "tie the stateid back to the client" using
rpcsec_gss.  All rpcsec_gss can guarantee is that the request was made
with certain credentials, but the credentials are tied to a gss
principal, not to a client.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 29 Aug 2005 18:22:13 -0400

On Mon, Aug 29, 2005 at 02:34:53PM -0700, Garth Goodson wrote:
> J. Bruce Fields wrote:
> >On Mon, Aug 29, 2005 at 04:20:30PM -0400, [email-masked] 

Why is that a problem?

--b.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 29 Aug 2005 15:57:02 -0700

>>This would just require the principal to match the stateid (as in the 

I didn't say it was a problem, just what needed to be done.

-Garth

---

From: Carl Burnett <[email-masked]>
Date: Mon, 29 Aug 2005 16:35:31 -0500

RFC3530  makes no requirements
on the client about a one to one relationship between credentials and open
state ids, so a server can assume this equivalence.  Also, with delegation,
the server would have to some how know of every credential that is valid
for access under the delegation stateid - something that does not sound
very possible since I don't think there is anything in the protocol that
allows the server to associate a cred in an ACCESS OP to a delegation state
id. That would be the ACCESS call required by the client for each "local"
open by a new user (cred) under a granted delegation.

Carl

Carl Burnett

AIX Kernel Architecture - Network File System

(512) 838-8498, TL 678-8498

(please reply to [email-masked])

"J. Bruce Fields"
<[email-masked]>

Sent by: [email-masked]

08/29/2005 03:59 PM

To

[email-masked]

cc

[email-masked]

Subject

Re: [nfsv4] pNFS, OPEN principals
and ACLs

Why not?

--b.

nfsv4 mailing list

[email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 29 Aug 2005 18:04:43 -0700

må den 29.08.2005 Klokka 16:35 (-0500) skreiv Carl Burnett:
> 
> RFC3530  makes no requirements on the client about a one to one
> relationship between credentials and open state ids, so a server can
> assume this equivalence.

Why should that stop pNFS from requiring it on the stateids that are to
be used on the data server?

Cheers,
  Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 30 Aug 2005 15:24:13 -0400

> be used on the data server?

Right.  I think not making such a requirement in 3530 was a
mistake--users shouldn't be able to close each others' opens!  But there
are two entirely separate questions here:

	1. Can we require nfsv4 clients to tie stateid's to principals?
	2. Can we make this requirement for nfsv4.1 clients sending
	   stateid's to data servers?

I don't know about #1, maybe it's too late for that.

But I can't see any objection to #2.

--b.

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 30 Aug 2005 14:38:22 -0500

On Tue, J. Bruce Fields wrote:
> > m? den 29.08.2005 Klokka 16:35 (-0500) skreiv Carl Burnett:
> > > 

Do we need errata or an interpretation draft for this?

Spencer

---

From: Carl Burnett <[email-masked]>
Date: Tue, 30 Aug 2005 18:32:36 -0500

If pNFS moves to generally require 1 to 1 relationships between state ids 
and credentials (users) for interaction with data servers, this has 
potential to greatly increase the state tracking footprints at both the 
client and the server. Think of 100,000 users accessing a collection of 
files from a small number NFS clients - for example, a WEB server securely 
offering proxy access to secured data residing in NFS. This could 
represent a scalability barrier. That is a trade-off to consider. The 
current model allows a client to manage all access by many users to an 
object with minimally 1 stateid under a granted delegation that was 
derived from a  given open owner. That's also just a 1 stateid drag on the 
server per client/file pair (after any related open and lock ids are 
relinquished). In the absence of a delegation, that minimally expands to a 
single open stated and, possibly, a lock stateid for each process doing 
file locking.

I don't view the current model as a mistake. It enhances the potential 
scalability of the protocol's state model and allows the client to 
contribute in scalable state management. The lack of a secure token in the 
RPC transactions that ties directly to a clientid instance is a 
deficiency. The potential good news is that Tom Talpey believes sessions 
can effectively fix that issue.

For the file based pNFS case, I wonder if this adds undo complexity by 
deviating from the existing RFC3530 requirements. I am not well studied on 
the specifics of the recent pNFS drafts, but its not clear how a client 
doing a normal OPEN, which allows a client to appropriately multiplex 
security (credentials) on an open owner with the server ensuring access on 
data operations, transitions to a model where owners and corresponding 
state expands down to the state per cred level. These specifics would need 
to be sufficiently described by the relevant spec (pNFS, NFS4.1, etc).

Presumably, a main motivation for pNFS is to efficiently move large 
amounts of data. If that's true, is the impact of data server access 
checks on each data transaction that critical? Especially, since its 
increasing the server side security.

Carl

Carl Burnett

"J. Bruce Fields" <[email-masked]> 
08/30/2005 02:24 PM

To
Trond Myklebust <[email-masked]>
cc
Carl Burnett/Austin/IBM@IBMUS, [email-masked], 
[email-masked], [email-masked]
Subject
Re: [nfsv4] pNFS, OPEN principals and ACLs

                 1. Can we require nfsv4 clients to tie stateid's to 
principals?
                 2. Can we make this requirement for nfsv4.1 clients 
sending
                    stateid's to data servers?

--b.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 30 Aug 2005 20:22:39 -0700 (PDT)

> can effectively fix that issue.

This is as good as time as any to talk about this ...

One of the problems with NFSv4.0 is vulnerability to a Denial of Service
attack via seqids.

Namely, the attack works via an unauthorized client faking an
open_owner4, an open_owner/lock_owner pair, or stateid, combined
with a seqid. The operation is sent to the NFSv4 server.
The NFSv4 server accepts the state information, and
as long as any status code from the result of this
operation is not NFS4ERR_STALE_CLIENTID, NFS4ERR_STALE_STATEID,
NFS4ERR_BAD_STATEID, NFS4ERR_BAD_SEQID, NFS4ERR_BADXDR,
NFS4ERR_RESOURCE, or NFS4ERR_NOFILEHANDLE, the sequence
number is incremented. When the authorized client issues
an operation, it gets back NFS4ERR_BAD_SEQID, because
its idea of the current sequence number is off by one.
The authorized client's recovery options are pretty
limited, with SETCLIENTID, followed by complete reclaim
of state, which may or may not succeed completely.
That qualifies as a denial of service attack.

Even if the data is exported with RPCSEC_GSS
protection, the attacker can use RPCSEC_GSS to make this attack work. If
he creates an RPCSEC_GSS context from a synthetic client, nothing
prevents him from constructing a COMPOUND that uses a valid
stateid and seqid for some operation. 

(I am assuming if the problem can be solved when using RPCSEC_GSS,
then the problem is solved; AUTH_SYS doesn't matter, because
AUTH_SYS is merely a large security hole.)

We could handle client and server behavior such that clients map
each open_owner and lock_owner to one any only one principal,
and servers enforce this. However not all clients are ready
for this, and Carl has noted the scalability issues that would
hinder implementing such behavior.

The NFSv4.1 sessions work provides an excellent architecture
for solving this. Namely, with sessions, there are no 
per operation seqids. There is a per
COMPOUND SEQUENCE operation tied to a real session and clientid.
Much simpler to implement, and as I'll claim in a bit, much easier
to secure.

The denial of service attack on sessions would need to forge 
the sessionid in a SEQUENCE operation in order to mess up the 
sequence number state. While possible based
on the sessions I-D (as currently written) such an attack is going to
be more obvious to an NFSv4.1 client and server. Still we can do better.

What we want to do is bind an RPCSEC_GSS context to a clientid. My proposal
is a BIND_CTX_TO_CLID operation that takes as input an RPCSEC_GSS context handle from
an establish RPCSEC_GSS context. BIND_CTX_TO_CLID is a new sessions op; it MUST be
preceded by the SEQUENCE operation, which in turn MUST be the first
op in a COMPOUND. The procedure MUST be invoked with RPCSEC_GSS integrity or
privacy. The principal that issues the COMPOUND with  
  { SEQUENCE, BIND_CTX_TO_CLID }
is that of the principal that created the clientid and session.

So what does this do? If an attacker creates an RPCSEC_GSS 
context and issues a COMPOUND with SEQUENCE and a valid sessionid, 
the server denies the access, because the context is not bound to 
the clientid. If the attacker spoofs a valid RPCSEC_GSS context 
handle, because he doesn't have the RPCSEC_GSS session key for
the clientid creating principal, he is denied.

So the denial of service described above is defeated.

What of the issue Garth raised? I propose that we introduce
two new operations, GETFD and PUTFD. Like GETFH, GETFD is invoked in the
same COMPOUND as OPEN, after OPEN. GETFD returns a file descriptor.
PUTFD injects a file descriptor. Unlike file handles, file descriptors
do not persist. One a file is CLOSEd, the file descriptor used
in the CLOSE is disposed of. PUTFD is used for any operation on
a regular file that takes a stateid. PUTFD is used instead of PUTFH.

If a client does a GETFD, the server returns the file descriptor, and binds
the principal of the user that issued the OPEN to the file descriptor. 
The server records in its memory the principal and the access rights (read, write, or
both) that the principal was allowed to do at the time of the OPEN.

Thereafter, use of the file descriptor on any operation on a
regular file does not require an access check. As long as the
principal matches, and the operation matches the access rights, the
operation is permitted.

GETFD can also be used after an ACCESS call. Thus if an OPEN that does
delegation is done, the client can issue an ACCESS request if a different
principal attempts to access the file internally. CLOSE is again used to dispose
of a file descriptor.

If we have file descriptors, why do we still need file handles? We need them
for caching. File descriptors don't last, whereas file handles, especially
persistent file handles, last a long time. Without file handles, it is not
practical to do long term caching of data.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 30 Aug 2005 21:58:02 -0700

> a regular file that takes a stateid. PUTFD is used instead of PUTFH.

I must be particularly dense today. Could you please be a bit more
explicit as to how this concept would differ from that of a filehandle
+stateid combination?

Cheers,
  Trond

---

From: Mike Eisler <[email-masked]>
Date: Tue, 30 Aug 2005 22:34:46 -0700 (PDT)

> explicit as to how this concept would differ from that of a filehandle

Stateids are shorthands for open owners and lock owners, where an open owner
and lock owner is not necessarily tied to one principal. Nor are the stateids
necessarily tied to a file, or an open file reference.  The stateid's purpose
is to indicate open, delegation and byte range lock state. Stateids are very
abstract, perhaps underspecified, which has led to at least two opposed,
but arguably both valid, interpretations of stateids.

(If we were doing a pure POSIX file access protocol, we'd might have simply used 
process IDs for stateids.)

A file descriptor is a shorthand for "the principal that opened the file".

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 30 Aug 2005 23:18:43 -0700

> but arguably both valid, interpretations of stateids.

Yes and no. Even RFC3530 appears to imply that a client should always
call OPEN and/or ACCESS in order to verify that it has sufficient
privileges to access the file. I don't see how anyone could interpret
this as being optional behaviour.
IOW: every stateid transition is supposed to be authenticated by the
metadata server, and it does not appear to be possible for a client to
just assume that it can just gratuitously switch to using a principal
that has not previously been authenticated by an OPEN.

The main argument in this thread appears rather to be whether or not a
server can assume that the client will associate a single principal to
each stateid, or if the server would have to track several principals in
order to map a stateid into an access.

> A file descriptor is a shorthand for "the principal that opened the file".

...but if you tie the lifetime to that of the stateid (you state that
the descriptor is only destroyed at CLOSE) how does that improve the
server's ability to cache authorisation information?

Couldn't you obtain similar performance if you use a scheme where the
server just pushes the principal for every successful OPEN (or if the
client has a delegation: for every ACCESS) to the data servers, which
then caches the result and then reaps the cache upon
CLOSE/OPEN_DOWNGRADE or SETATTR?

Cheers,
  Trond

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 31 Aug 2005 01:36:35 -0500

> a regular file that takes a stateid. PUTFD is used instead of PUTFH.
...

Sounds close to an access token; why not deal with NFS as the backend
protocol as with an OSD...  The client could get the FD from the
metadata server and then PUTFD the same thing on the data servers.  To
build a token out of this is easy, just structure it something like,
roughly:

{ FD-data = {issuer, fd-index-local-to-issuer, expiration, FH, princ name,
	     access-type},
  key-id, HMAC(K, <FD-data>) }

(public key crypto would be better, but slower)

First time validation involves validating the MAC, subsequently you can
cache the tokens and index them by {issuer, fd-index-local-to-issuer},
memcmp the FD with the cached value and check that the RPCSEC_GSS
context handle used has the right client principal name.

This gets the data servers out of having to eval ACLs too, just as with
an OSD.

Nico
--

---

From: Carl Burnett <[email-masked]>
Date: Wed, 31 Aug 2005 10:22:32 -0500

Hi Mike,

The getfd/putfd idea sounds interesting. Could the getfd be something that 
the server optionally supplies? The thought is that if the server has 
efficient means to quickly determine access from only the filehandle and 
auth_context, it might want opt out of supplying the fd to the client and 
just do its normal access processing on the i/o ops. Also, the server may 
have security policies beyond the basic access controls on the file - 
things like time of day restrictions, etc. Similarly, the client's use of 
GETFD might also be optional.

While this might be pretty cool, In the big picture, this seems like a 
lower priority item. Its a performance tweak.

Sessions has perhaps emerged as the most important feature on the table 
right now. It seems it offers the broadest set of potential benefits (DOS 
protection, performance/scaling efficiencies, exactly once semantics), and 
it seems to be often an assumed prereq for other NFS features (pNFS, 
optimized RDMA). I am a big proponent of directory delegation, but I would 
have to concede that sessions looks more critical for advancing the state 
of NFSv4 near term.

Maybe if NFS 4.1 were limited to only sessions, it could reach consensus 
quicker and therefore show up in products sooner. Then there would be a 
foundation for bringing in the other features. To make it easier to bring 
in follow-on features, maybe 4.1 could also include some type of generic 
plug-in facility that allows dynamic insertion of new NFS features without 
requiring a formal rev of the NFS protocol itself.

Just a thought,
Carl

Carl Burnett

Mike Eisler <[email-masked]> 
Sent by: [email-masked]
08/30/2005 10:22 PM
Please respond to
email2mre-ietf

To
[email-masked]
cc

Subject

What we want to do is bind an RPCSEC_GSS context to a clientid. My 
proposal
is a BIND_CTX_TO_CLID operation that takes as input an RPCSEC_GSS context 
handle from
an establish RPCSEC_GSS context. BIND_CTX_TO_CLID is a new sessions op; it 
MUST be
preceded by the SEQUENCE operation, which in turn MUST be the first
op in a COMPOUND. The procedure MUST be invoked with RPCSEC_GSS integrity 
or
privacy. The principal that issues the COMPOUND with 
  { SEQUENCE, BIND_CTX_TO_CLID }
is that of the principal that created the clientid and session.

If a client does a GETFD, the server returns the file descriptor, and 
binds
the principal of the user that issued the OPEN to the file descriptor. 
The server records in its memory the principal and the access rights 
(read, write, or
both) that the principal was allowed to do at the time of the OPEN.

GETFD can also be used after an ACCESS call. Thus if an OPEN that does
delegation is done, the client can issue an ACCESS request if a different
principal attempts to access the file internally. CLOSE is again used to 
dispose
of a file descriptor.

If we have file descriptors, why do we still need file handles? We need 
them
for caching. File descriptors don't last, whereas file handles, especially
persistent file handles, last a long time. Without file handles, it is not
practical to do long term caching of data.

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 31 Aug 2005 12:06:18 -0500

> single open stated and, possibly, a lock stateid for each process doing 

As with all things in life, the issues are multi-dimensional.

With respect to the choice of openowner/stateid scope at the client...

It is certainly true that the client gets to choose the model it wants
for openowner scope and in doing so can reduce the burden at the client
and server for the memory footprint of state management.  However, choosing
a broad scope for the openowner will increase the serialization impact
of sequence id transitions for a workload that has a lot of open/close
transitions.

Our experience with Solaris shows that the serialization of the sequence
id space can have a significant impact.  THe Solaris client is currently
implemented to have an openowner for each credential pointer within the
client.  This means that a user's process, sharing the same credential
pointer, will have the same openowner for a mount in Solaris.  
Our measurements of workloads that are multi-process(threaded) in nature
and have this high degree of open/close are impacted by the additional
latency of the sequence id management.  The easiest example is a
large N-way software build.  We are working on strategies to keep the 
scope of the openowner but overcoming through implementation the sequence id
problems.

Spencer

---

From: Carl Burnett <[email-masked]>
Date: Wed, 31 Aug 2005 13:36:03 -0500

Hi Spencer,

As you say, many trade-offs to consider. An n^3 state footprint model on 
the server with higher volumes of OPEN/CLOSE transactions and reduced 
delegation benefit sounds like a scary proposition for NFSv4 scalability 
to me. A big price to pay for some performance gain with a single client 
to server workload.

Especially if you consider that processor power just keeps increasing and 
sessions can obviate the need for sequencing on the lower level NFS state 
anchors.

Carl

Carl Burnett

Spencer Shepler <[email-masked]> 
Sent by: [email-masked]
08/31/2005 12:06 PM
Please respond to
spencer.shepler

To
[email-masked]
cc

Subject

and server for the memory footprint of state management.  However, 
choosing

scope of the openowner but overcoming through implementation the sequence 
id
problems.

Spencer

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 31 Aug 2005 14:40:03 -0400

On Wed, Aug 31, 2005 at 01:36:03PM -0500, Carl Burnett wrote:
> the server

Sorry, I've lost track--what's n here?

--b.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 31 Aug 2005 12:02:09 -0700 (PDT)

> metadata server, and it does not appear to be possible for a client to

Where does it say that?

> that has not previously been authenticated by an OPEN.

I agree that the principal you open the file with is the principal you
ought to be doing the I/O with.

> server's ability to cache authorisation information?

There is no "the" stateid. There are multiple stateids. Each open file has one for
the openowner, and one for each process that is forked and doing byte range
locking.

> CLOSE/OPEN_DOWNGRADE or SETATTR?

I think that's more or less what I'm proposing, but I'm introducing a new
concept that isn't cluttered with all the process ID stuff on the client.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 31 Aug 2005 12:08:45 -0700 (PDT)

> Sounds close to an access token; why not deal with NFS as the backend

It is solving the same problem as the access token, but it isn't an access token.

> (public key crypto would be better, but slower)

Where does key K from from? A private protocol between the metadata and
date servers? I prefer securing this via the open protocol rather than trusting
the implementation to secure the private protocol.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 31 Aug 2005 12:21:54 -0700 (PDT)

> the server optionally supplies? The thought is that if the server has 

Carl,

I think if pNFS w/ file layouts is supported by the server, then GETFD and PUTFD
need to be mandatory operations. 

> auth_context, it might want opt out of supplying the fd to the client and 

File handles as capabilities or access tokens are another way to achieve what
file descriptors do. However, such file handles must by volatile, and a drawback
of volatile file handles is that the NFS client cannot usefully cache data associated
with the file handle once the file handle has expired.

> lower priority item. Its a performance tweak.

It's a way to solve the problem Garth brought up. If by performance tweak you mean
that the data server can always send ACCESS checks to the metadata server on
each I/O, sure, but if that's what the data server has to do, then whither pNFS.
Another way to do this in addition to what Nico and you have suggested, is
for the data servers and meta data servers to have a private protocol that
gets delegation or notification on each open file's ACL. Effectively mandating
complex private protocols I think would impede pNFS implementation progress.

> requiring a formal rev of the NFS protocol itself.

I hear you, but wonder if implementors are keen to implement sessions without
a visible shiny new feature.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 31 Aug 2005 20:07:45 -0500

> It is solving the same problem as the access token, but it isn't an access token.

But without a metadata server<->data server protocol the FD is not
enough.

> the implementation to secure the private protocol.

It could be a long-term key shared between each metadata and data server
pair, say.  That'd take no more configuration than it would take to
establish trust between them for direct communications between them.