Subject: [nfsv4] Client state security and the SSV

From: Mike Eisler <[email-masked]>
Date: Sat, 21 Apr 2007 00:14:17 -0700 (PDT)

Trond raised an issue with the sessions inspection team in his
written comments, and in the part 2 of the inspection a few days
ago. His written comments are:

- According to para 2.10.6.3 (line 2979) the 
   SSV/BIND_CONN_TO_SESSION stuff is supposed to 
   protect against someone hijacking a given session, and
   messing with the locks/delegations/...

   What prevents the attacker from guessing the clientid, 
   and then opening a new session using CREATE_SESSION? 
   AFAICS that would allow it to issue LOCK or CLOSE commands 
   on behalf of the client since the lease and locking state 
   are tied to the client ID, not the session according to
   line 2161.

   IOW: Why is the SSV tied to the session, and not to the clientid?

--------------------------------------------

So Trond is correct that as specified, an attacker could guess
the clientid, open a new session, and issue LOCK or CLOSE
operations which would then mess up the legitimate client.

With respect to his question, why is the SSV tied to the session
and not the clientid, the answer is at the time I did not think
it was feasible. Here were the obstacles I saw:

- The SET_SSV operation XORs the argument with the server's SSV.
  By relying on the exactly once semantics of sessions, it is trivial
  to keep the client and server synchronized on what the SSV is.
  Still, one could pronounce that SET_SSV is tied to the client,
  and nonetheless MUST be issued inside a session (i.e. in a COMPOUND
  that starts with SEQUENCE). 

- At the time I conceived the SSV, the spec said a
  client ID MUST be shared across the metadata server 
  and data servers. A common SSV was thus
  a non-starter. We fixed that.

- The NFSv4.1 server supports client ID trunking which means that
  a client ID can be used against two different server network
  addresses, which
  map to the same logical server,  but are backed by different
  physical controllers which have a higher latency interconnect
  that say a server with symmetric multiprocessors sharing the
  memory address space. Thus for reasons of server architecture/
  topology, the server requests separate sessions (possibly because
  the overhead of updateding the sequenceid on a slot across each
  server network address is too expensive. The obstacle I saw
  was that the cost of keeping the SSV synchronized across 
  this higher latency, intra server link was impractical. However,
  the specification requires two different sessions sharing the
  same client ID to allow OPEN/LOCK/LOCKU/CLOSE/etc. operations
  to work independent of the session. Surely, OPEN will happen
  more frequently than SET_SSV.

So I think Trond is correct; I now support tying the SSV to the
client ID.

Also, note that DESTROY_CLIENTID is now part of the spec.
The spec says nothing about which principal can issue DESTROY_CLIENTID.
It does say that DESTROY_CLIENTID fails if there is any state
associated with the client ID. We could allow anyone to issue
DESTROY_CLIENTID, with little consequence since other than the
client ID, nothing is lost. Or we could add an SSV digest to
DESTROY_CLIENTID (since the SSV would be per-clientid). 
Spencer points out that I failed to add
DESTROY_CLIENTID to the operations struct in the .x file, so
technically, DESTROY_CLIENTID is not part of NFSv4.1 yet. So
changing DESTROY_CLIENTID is not a .x change (in a weasely
kind of way :-).

I was thinking of proposing that we require that CLOSE, LOCK, LOCKU,
etc. be done with the same credential that issued OPEN instead of
upgrading the SSV. That is not necessary to solve the problem
Trond identified. Still, it is something that I think we should
consider negotiating when EXCHANGE_ID is issued.

The inspection of the sessions section (but not the individual
operations; that is to be scheduled as a separate, but related
inspection effort) is concluded, so I plan to start making
the edits, based on the comments, and unless there are 
objections, change the SSV to be
per-client ID and add an SSV digest to DESTROY_CLIENTID.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Rick Macklem <[email-masked]>
Date: Mon, 23 Apr 2007 11:42:05 -0400 (EDT)

On Sat, 21 Apr 2007, Mike Eisler wrote:
[stuff snipped]
> consider negotiating when EXCHANGE_ID is issued.
What about the following case:
- "rick" opens file "foo" for writing, gets OpenStateId-foo using "rick"
     credentials
- process forks off long running child process to run in background
   - this process inherits open of "foo"
   - this process does a setuid "daemon" (which has credentials in a keytab
       so they don't expire during the long run)
- "rick" logs out and "rick" credentials go away
   - child process terminates, doing a Close of OpenStateId-foo

For a case like this, requiring Close to use the same credentials as the
Open would force the client to re-open all files when a setuid occurs,
using the new credentials ("daemon" for the above case). Suppose "daemon"
doesn't have permissions to write "foo"? (This might be reasonable, since
"daemon" is some generic credentials in a keytab, used for long running
processes and might not have access.)

It seems the the big change between RFC3010 and RFC3530 w.r.t. state
handling was for cases like the above. Requiring the Close to use the
same credentials as Open, seems to break it?

rick

---

From: Mike Eisler <[email-masked]>
Date: Mon, 23 Apr 2007 09:01:30 -0700 (PDT)

>    - child process terminates, doing a Close of OpenStateId-foo

I don't know about FreeBSD and other kernels, but last I checked
Solaris, the "ucred" of the user that opened the file and generated the
file descriptor is get with the open descriptor. All subsequent
operations, even through the vnode layer, use the "ucred" assigned
to the open file descriptor.

If that wasn't the case, then given that NFSv3 has no OPEN operation,
how would it work for user joe to open a file he owns with mode
bits 0600, run a setuid process to steve, and then that process tries
to issue I/O?

> using the new credentials ("daemon" for the above case). Suppose

No, because under the scenario I gave, re-opening the file for steve
would not work. The open file descriptor has joe's credentials,
so issuing the CLOSE with joe's credential is quite feasible.

> "daemon"

The state handling in RFC3510 and the security was underspecified.
It was a surprise to be that some clients were handling the
joe/steve scenario differently that Solaris.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 23 Apr 2007 11:16:00 -0500

On Mon, Apr 23, 2007 at 09:01:30AM -0700, Mike Eisler wrote:
> > - "rick" opens file "foo" for writing, gets OpenStateId-foo using
> >   "rick" credentials
> >    - this process does a setuid "daemon" (which has credentials in a
> >      keytab so they don't expire during the long run)

Solaris has done this for a looong long time -- file_t has a reference
to the cred_t of the process (thread) that originally opened the file,
and all operations on any file_t are done with those credentials, not
with the credentials of the calling process.  Unix semantics require
this (if nothing else because there was no error to describe the "user
that opened this fildes has logged off" prior to the advent of networked
file systems).

> > For a case like this, requiring Close to use the same credentials as
> > the Open would force the client to re-open all files when a setuid
> > occurs, using the new credentials ("daemon" for the above case).
> > Suppose
> 

Quite.

> 
> > "daemon" doesn't have permissions to write "foo"? (This might be
> > reasonable, since "daemon" is some generic credentials in a keytab,
> > used for long running processes and might not have access.)
> > 

IIRC there is some standard SUS or POSIX requirement w.r.t. this.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 23 Apr 2007 12:48:52 -0400

I think that when we address this issue, we address all of it.

Some areas that havn't been mentioned, so far:

     Credential identity requirement for open upgrade.

     Credentials for open-downgrade.

     Credential requirements, if any, for delegation-return.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Monday, April 23, 2007 12:02 PM
To: [email-masked]
Subject: What credentials should a setuid process use after fork (was
Re:[nfsv4] Client state security and the SSV)

--- Rick Macklem <[email-masked]> wrote:

> "rick"
> keytab

> the

---

From: Rick Macklem <[email-masked]>
Date: Mon, 23 Apr 2007 13:13:05 -0400 (EDT)

On Mon, 23 Apr 2007, Mike Eisler wrote:
[stuff snipped]
> to the open file descriptor.
Yep, and BSD does the same. I hadn't meant my example to be BSD specific.
(I suppose, in retrospect, it was a crummy example:-)

Having said that, it's not obvious to me that it is the correct thing to
do for NFSv4. My server currently allows any credential to do the Close
on a given StateId. (You already mentioned under specified.) If I enforce
the "same credential as Open" rule and a client no longer has a valid
credential for the user that did the Open because it has expired, then
my server gets stuck with a bunch of Open state which it can't throw away
because the client's Close failed.

And what happens if the same happens for Locks? The server ends up with
a bunch of Locks that the client can't LOCKU, which will cause grief,
sooner or later.

Did I make my concern clearer this time? rick
ps: To be honest, I had never looked to see whether BSD used the file
     descriptor credential or process/thread credential, for close. It
     does do the formal, just like read/write.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 23 Apr 2007 12:20:26 -0500

> because the client's Close failed.

The client behaviour should be to use the same credential for all ops on
an open file descriptor, but that doesn't mean that the server should
enforce this -- the client does this for reasons of its own (POSIX
semantics), not because the server demands it.

Nico
--

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 23 Apr 2007 12:38:35 -0500

On Apr 21, 2007, at 2:14 AM, Mike Eisler wrote:

<details elided>
> per-client ID and add an SSV digest to DESTROY_CLIENTID.

I support these changes.

Spencer

---

From: Mike Eisler <[email-masked]>
Date: Mon, 23 Apr 2007 10:51:16 -0700 (PDT)

--- Nicolas Williams <[email-masked]> wrote:

> an open file descriptor, [...]

So if the client uses the same cred for all operations as that
recorded in an open file descriptor, then even ignoring setuid/setgid
scenarios, once a user's Kerberos credentials expire, doesn't
this mean the client doesn't even get to attempt
the CLOSE? Because the client cannot even get an RPCSEC_GSS context
in order to the issue the CLOSE with? So I don't see how Rick's
literal expression of the scenario, the "server gets stuck with a bunch
of Open state which it can't throw away because the client's Close
failed" is ever reached. The server gets stuck with a bunch
of OPEN state for sure, but it doesn't get thrown away because
the client is RENEWing successfully, but not even sending CLOSEs.

Perhaps clients in this situation use AUTH_SYS
for CLOSEs, even if the filesystem is exported with only sec=krb5.
Not good.

RFC3530 discusses a somewhat related problem ... that of
uncommitted, unwritten data in the client's cache on filehandles
protected with non-AUTH_SYS security flavors:

   With respect to authentication, flushing modified data to the server
   after a CLOSE has occurred may be problematic.  For example, the
user
   of the application may have logged off the client and unexpired
   authentication credentials may not be present.  In this case, the
   client may need to take special care to ensure that local unexpired
   credentials will in fact be available.  This may be accomplished by
   tracking the expiration time of credentials and flushing data well
in
   advance of their expiration or by making private copies of
   credentials to assure their availability when needed.

I.e. spec allows no weaseling out for WRITEs when the Kerberos creds
expire. Why should CLOSE be any different?

In other words, the onus is one the client to keep
unexpired creds around to clean up his crap.

In the case of OPENs done with a Kerberos credential, once the
cred expires on the server, then the answer to the spirit of 
Rick's server dilemma is:  revoke the OPEN when necessary. When
necessary means that a conflicting OPEN occurs, or the server is out
of resources.

> semantics), not because the server demands it.

If the the server should not enforce it, then why should the
client use the same credential?

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 23 Apr 2007 12:59:25 -0500

> the CLOSE? Because the client cannot even get an RPCSEC_GSS context

No, I said "should" not "must."

The issue is that the client normally wants to preserve POSIX semantics,
but has to deal with credential expiration.  Closing the file and
returning EIO (or whatever is appropriate) to the caller is fine.

> [...]

> client use the same credential?

I answered that in the text you quote.   Some clients may not care for
such semantics, in which case they are free to do as they please.

Nico
--

---

From: Mike Eisler <[email-masked]>
Date: Mon, 23 Apr 2007 11:13:04 -0700 (PDT)

--- Nicolas Williams <[email-masked]> wrote:

> > > The client behaviour should be to use the same credential for all
> ops
> > > on

Then let's get to the more relevant part: what are clients in this
situation doing today?

> returning EIO (or whatever is appropriate) to the caller is fine.

What operation (that returns EIO) does the client issue, 
and with what credentials and security 
flavor is that operation issued with?

> such semantics, in which case they are free to do as they please.

If some clients don't care, and the server's therefore don't care,
then why should the other client's care?

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 23 Apr 2007 13:28:15 -0500

> Then let's get to the more relevant part: what are clients in this

Dunno.

> flavor is that operation issued with?

The client could do any number of things.  It could wait, and wait, and
wait for credentials to appear locally (i.e., the user to log back in).
It could CLOSE the file with its own or the calling process'
credentials.  It could wait for a while and then do that.  It could
crash :)

> then why should the other client's care?

What other clients?

---

From: Mike Eisler <[email-masked]>
Date: Mon, 23 Apr 2007 11:42:31 -0700 (PDT)

--- Nicolas Williams <[email-masked]> wrote:

> wait for credentials to appear locally (i.e., the user to log back

So no EIO.

> in).

The creds have expired. Most GSS-API implementations I'm aware of
do not return a context from gss_init_sec_context(). I suppose
the client could issue the NFS request with an old, expired,
context, but the server is then going to return an RPC/RPCSEC_GSS
authentication error, not an EIO.

The calling process will normaally have the same effective
uid. As I pointed out, we have a problem with expired
creds whether setuid is involved or not, and so I simplified
the problem. Perhaps I should change the email subject again. :-)

So I'm not seeing how the server ever returns EIO. All I can
see is that the server knows when the RPCSEC_GSS context has
expired, and it can use the expiration to do "administrative"
revocation of OPENs when necessary.

> > > I answered that in the text you quote.   Some clients may not
> care
> > > for

There are two types of clients: those that always issue post-OPEN
operations using the same credential as the OPEN was issued
with and those  that don't always do this. Let's call
theses types A and B respectively. There are two types
of servers, those that enforce type A behavior and those that
allow type B behavior. Let's call these types of servers
C and D.

So to restate:

If there exist type B clients, and there do not exist type C
servers, then why should type A clients exist? Why should
a client implementor bother going the type A route, if type
C servers are not permitted to exist?

---

From: Rick Macklem <[email-masked]>
Date: Mon, 23 Apr 2007 14:44:10 -0400 (EDT)

On Mon, 23 Apr 2007, Mike Eisler wrote:
> in order to the issue the CLOSE with?
Why can't the client use some other credentials if the one used for the
Open has expired. For example, it could use the same credentials it is
allowed to use for Renew. (Those are specified in RFC3530 Sec. 14.2.28.)
If a client can Renew state with a given principal, shouldn't it be able
to close state with that principal, as well?

Alternately, for 4.1, the credentials used to privacy protect the SSV
could be allowed. It appears that a client has to maintain these in
a current state by using a machine principal or establishing a new SSV
for each principal used. (Bottom of Pg. 55 of Draft 10.) 
If I understood Trond's recent suggestion of associating the principal
with the ClientID instead of session, I would say allowing this alternate
case imply that the principal should be associated with a session, since
it is easier to create a new session (for the non-machine credential 
case) to establish the new principal as the "current one".
> expire. Why should CLOSE be any different?
Because Close requires the server to maintain state, so long as the
client keeps doing Renews (see above w.r.t. using same credentials for
Close as allowed for Renew).

> unexpired creds around to clean up his crap.
Which generally works for Writes, because those are flushed to the server
within seconds. Solaris gives a 30min warning of credential expiration,
as I recall. (I don't know if it switches to write-through caching at that
time, but could do so if it wished to.)

If a client does a Close before the associated process/thread perfroms the
close system call, then it has to somehow re-open the file, if the
process/thread uses the descriptor again. Yuck.

> necessary means that a conflicting OPEN occurs, or the server is out
Revoking state is, to me, something that should be avoided whenever
practicable. The client recovery once state is revoked is primitive at 
best. To require that just because a principal on a client has expired,
doesn't seem appropriate to me.

>> semantics), not because the server demands it.
I'll agree with Nico on this one.

> client use the same credential?
Personally, if it going to be more restrictive than any valid principal,
the requirement should be "the same as required for Renew".

rick

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 23 Apr 2007 13:47:08 -0500

When I mentioned EIO I was talking about what the client returns to the
application, not what the server returns to the client.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 23 Apr 2007 12:07:44 -0700 (PDT)

> to close state with that principal, as well?

Yes, I think for NFSv4.0 this would/should/must be acceptable.

> could be allowed. It appears that a client has to maintain these in

An infinite number of creds to privacy protect
SET_SSV are possible, including those of Tom, Dick, and Harry.
But I don't believe Harry wants Tom CLOSEing Harry's files.

> case imply that the principal should be associated with a session,

No, NFSv4.1 removes all one to one association between clientid and
a principal. This was what the WG wanted. 

I agree these problems are much simpler to solve with machine creds,
but
the complaints from customers and implementors are very loud and
numerous.

We could have yet another GSS-mechanism that uses the SSV for
producing contexts. Personally I've got mechanism fatigue.
There is one true mechanism and it is called Kerberos.
So we need to be figure out to solve things on the assumption all
we have is Kerberos.

> as I recall. (I don't know if it switches to write-through caching at

This warning is only sent to users, AFAIK.

Nothing prevents a client from sorting his RPCSEC_GSS context
list by order of next to expire. It isn't rocket science.

> process/thread uses the descriptor again. Yuck.

Kerberos creds have expiration times. I don't know
of a general solution to the problem. All creds from all
mechanisms can expire. But Kerberos tells the client and server
when something is nearly expired, and when it has expired.
So clients and servers have all the tools needed to keep
from 

> practicable. The client recovery once state is revoked is primitive

I agree. But we have conflicting OPEN or the server is out of
resources. The client can't get Kerberos creds for Tom, Dick, Harry,
or whoever it issued the SETCLIENTID/EXCHANGE_ID with. What is
the server going to do? Allow CLOSEs with AUTH_SYS creds?

> at 

I disagree; it is totally appropriate if there is no other 
(secure) solution.
> I'll agree with Nico on this one.

I'd like to understand why.

The POSIX semantic exists for a reason: secure operation.
Either the client should have assurance from the server it will
enforce it, or we should not bother.

> the requirement should be "the same as required for Renew".

That takes care of NFv4.0.

---

From: Rick Macklem <[email-masked]>
Date: Mon, 23 Apr 2007 16:12:09 -0400 (EDT)

> But I don't believe Harry wants Tom CLOSEing Harry's files.
Hmm. I think that most clients will one of two types.
1 - Shared among several users. For these a machine credential in
     a keytab seems appropriate. (So, I think a machine credential
     should be an acceptable alternative to the user credential that
     Open'd the file.)
OR
2 - A single user machine. In this case, everything gets done using
     the one credential, so that case seems fine, too. (That's how my
     current client port for Mac OS X works.)

However, I can see others wanting a multi-user machine without
machine credentials. To allow that case, the server must "trust"
the client enough that it is willing to let "Harry" close "Fred's"
files, knowing that it is doing so because "Fred's" process wants to,
but Fred has taken a long lunch and his TGT has expired.

So, I can see allowing the above as a server per-host option.
(In this case, the "trust" isn't as strong as allowing AUTH_SYS,
  since it requires the client to have some valid kerberos credential.
  Also, a premauture close could cause an app. to fail, but not data
  corruption.)

> resources.
If, by conflicting OPEN, you mean a Share/Deny conflict, I'd say that
it's going to be unlikely, since most (all?) POSIX clients use DENY_NONE.

As for resource exhaustion, how does the server know what to Revoke?
ie. Which Opens are lying about because the client's credential expired
vs the ones where the client doesn't want to Close yet. The server might
know that certain credential handles have expired, but it doesn't know
if the client has a renewable TGT for those ones, that it just hasn't
needed to renew (in the Kerberos sense) yet.

If the same rule applies to Close as it does Renew, then revoke isn't
necessary, since the state will expire, because the client can't Renew
(NFSv4 sense) either.

rick

---

From: Mike Eisler <[email-masked]>
Date: Mon, 23 Apr 2007 14:15:12 -0700 (PDT)

In thinking about this more, while I disagree with the 
assertion that Fred should be allowed to CLOSE Harry's files, even
if Harry or Harry's client disagrees, I also see value in
Rick's insight that the machine cred that created the client ID
via EXCHANGE_ID should be allowed to CLOSE the Harry's files.
The one trouble with that is that in a multi-user machine with
no machine creds, Fred might have been the one that issued the
EXCHANGE_ID.

So I'm thinking that we need a way to involve the SSV in
protecting CLOSE, LOCKU, etc. to allow the client to
tear down state of users that have expired creds.

A new GSS mechanism that uses the SSV as an RPCSEC_GSS
authenticator is obviously one way to do it, but
I don't believe in new mechanisms any more. I'll think on it
more.