Subject: Re: [nfsv4] Problem statement: Dealing with compromised NSDBs

From: Chuck Lever <[email-masked]>
Date: Mon, 3 Jun 2013 15:54:16 -0400

Summarizing comments:

  o  A compromised NSDB can be dealt with by implementation choices,
     but there would be no way to address it via the ADMIN protocol

  o  The Working Group chair prefers a -bis or ADMINv2 document to
     resolve this.  The current WG charter covers this work

  o  A fix will require XDR alterations; there are few ADMIN
     implementations in the wild, so we can accommodate this change
     via a -bis rather than v2

  o  The Area Director prefers an ADMINv2 document

  o  A narrowly-scoped ADMINv2 could serve as a bug-fix release, and
     future work could be relegated to ADMINv3

  o  Do we have adequate (expert LDAP) reviewers for this change?

The question for the Working Group is how this should proceed:

  1.  An ADMIN-bis document to specifically address this problem

  2.  An ADMINv2 document to specifically address this problem

  3.  An ADMINv2 document to address this problem and accommodate
      future FedFS work

On Apr 9, 2013, at 12:45 PM, Chuck Lever <[email-masked]> wrote:

> Hi-
> 
> The FedFS ADMIN RPC protocol provides a mechanism for provisioning NSDBs on remote fileservers. The operations it provides are FEDFS_SET_NSDB_PARAMS, FEDFS_GET_NSDB_PARAMS, and FEDFS_GET_LIMITED_NSDB_PARAMS.
> 
> FEDFS_SET_NSDB_PARAMS specifies the name of an NSDB and the security mode to use when connecting to this NSDB.  The fileserver connects to an NSDB in order to resolve a FedFS junction.  The ADMIN protocol specification further says:
> 
>> On success, this operation returns FEDFS_OK.  When the operation
>> returns, the new connection parameters SHOULD be used for all
> 
>  [ Btw: When would a fileserver choose not to use these connection
>    parameters?  I think that SHOULD ought to be MUST, otherwise
>    an administrator might specify a secure connection mode but get
>    no security. ]
> 
>> subsequent LDAP connections to the given NSDB.  Existing connections
>> MAY be terminated and re-established using the new connection
>> parameters.  The connection parameters SHOULD be durable across
>> fileserver reboots.
> 
> 
> There are two security modes defined in the protocol specification:  FEDFS_SEC_NONE, which does not authenticate the LDAP server; and FEDFS_SEC_TLS, which uses START_TLS (RFC 4513) to authenticate the LDAP server.
> 
> When FEDFS_SEC_TLS is specified with the FEDFS_SET_NSDB_PARAMS operation, an x.509v3 certificate chain is provided to the fileserver, which it uses to authenticate subsequent connections to this NSDB.  FEDFS_SET_NSDB_PARAMS can change the connection security used by a fileserver to connect to a particular NSDB from NONE to TLS or TLS to NONE.
> 
> The idea is that over time, domain administrators will add NSDB connection parameters to each of their fileservers to allow FedFS junction resolution.  The specified NSDB may be the domain's own, or it might be an NSDB in a remote domain.
> 
> Many junctions on multiple fileservers can be created that use a particular NSDB.  There is no way to find such junctions without an exhaustive search.  Since filesystem namespace topology can evolve arbitrarily over time, a recorded pathname of any junction cannot be relied upon.
> 
> Now suppose we have two FedFS domains: example.net and university.edu.  Suppose university.edu fileservers have a number of junctions that refer to locations maintained by example.net, and thus use example.net's NSDB.
> 
> One day Mallory compromises example.net's NSDB, but the domain administrator there is on a long vacation.  The administrator at university.net discovers the compromise immediately, but has no control over the remote NSDB and cannot create a fresh x.509 certificate or verify that the contents of the NSDB are unmolested.  The only choice is to find and remove every junction in the university.edu that contains the compromised NSDB.
> 
> If university.edu is using a good implementation of FedFS, the command line tools it provides might allow the administrator to simply visit each of its fileservers and mark the example.net NSDB as compromised.  Any junction resolution that attempts to use that NSDB would fail, but the junctions remain in place.  When example.net's administrator gets back from holiday and cleans up the mess, the university.edu administrator can then update each of her fileservers with fresh connection parameters for that NSDB.
> 
> However, none of this can be done remotely using the FedFS ADMIN protocol.  It does not have a mechanism for removing NSDB connection parameters or for fencing a compromised NSDB.  This gap seems apparent to the audience whenever I give a talk about FedFS.
> 
> Here are some naive possibilities for addressing this:
> 
> 1.  Add another connection security mode called, say, FEDFS_SEC_FENCED or FEDFS_SEC_REVOKED.  Subsequent fileserver connections to the NSDB fail with FEDFS_ERR_NSDB_AUTH.
> 
> 2.  Add a new ADMIN protocol operation that removes an NSDB from a fileserver's NSDB connection parameters database.  Fileserver connections to any NSDB not in the fileserver's NSDB connection parameters database fail with FEDFS_ERR_NSDB_PARAMS.
> 
> 
> Analysis:
> 
> 1. would require a smaller change to the ADMIN protocol specification, which is an advantage given that the document is now in the RFC editor queue.  It would be a matter of adding the new value in the XDR definition of FedFsConnectionSec, and a new paragraph in the FedFsNsdbParams description in chapter 4 which discusses the meaning of FEDFS_SEC_FENCED and any behavior requirements.
> 
> Changing an NSDB's connection parameters from FEDFS_SEC_REVOKED to FEDFS_SEC_TLS would require the domain administrator to re-present the NSDB's x.509 certificate to a fileserver.  Our preference might be to insist on a certificate as a safeguard, or to keep the old one as a convenience.
> 
> This setting might be a little confusing for NSDBs that used to be in FEDFS_SEC_NONE mode, which require no authentication, and thus FEDFS_ERR_NSDB_AUTH might be unexpected.  An alternative would be to add a specific error code, say, FEDFS_ERR_NSDB_FENCED, to report this security setting when a connection to this NSDB is attempted.
> 
> In order for this to work securely, the SHOULD in the quotation above will have to be changed to MUST.  Otherwise, a fileserver can choose to continue connecting to an NSDB that the administrator wants fenced.
> 
> 
> 2. would be a somewhat larger overhaul, which could be a problem for finishing the ratification of the ADMIN protocol specification.  It might be better to leave this kind of change for an ADMIN v2 protocol.
> 
> I'm not sure what kind of changes are permitted at this stage.  If we can't make any protocol changes, then our only choice might be to embark on ADMIN v2 if we deem this gap important.
> 
> -- 
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Nico Williams <[email-masked]>
Date: Mon, 3 Jun 2013 15:52:13 -0500

>     an administrator might specify a secure connection mode but get

[ Seems like a requirement that the operation return be synchronous
with the configuration update. ]

---

From: Nico Williams <[email-masked]>
Date: Mon, 3 Jun 2013 16:02:09 -0500

> One day Mallory compromises example.net's NSDB, but the domain administrator there is on a long vacation.  The administrator at university.net discovers the compromise immediately, but has no contr…

Actually, the university.edu admin *can* *always* generate a
fresh-but-bogus certificate (heck, we could publish a suitable bogus
cert to use for this; should be trivial) for the example.net NSDB,
update the fileservers at unversity.edu, and let all clients of the
example.net NSDB fail to authenticate it thereafter.  This is, I
think, the correct course of action.  It still requires enumerating
all the fileservers at university.edu, but not all junctions.

> If university.edu is using a good implementation of FedFS, the command line tools it provides might allow the administrator to simply visit each of its fileservers and mark the example.net NSDB as…

Right.

> However, none of this can be done remotely using the FedFS ADMIN protocol.  It does not have a mechanism for removing NSDB connection parameters or for fencing a compromised NSDB.  This gap seems a…

It might be useful to have a DB for each site that lists the NSDB
security parameters for federated NSDBs, then the "revocation"
operation need only take place in one place per-site.

> 1.  Add another connection security mode called, say, FEDFS_SEC_FENCED or FEDFS_SEC_REVOKED.  Subsequent fileserver connections to the NSDB fail with FEDFS_ERR_NSDB_AUTH.

This would be nice, but see above -- we can workaround not having this.

> 2.  Add a new ADMIN protocol operation that removes an NSDB from a fileserver's NSDB connection parameters database.  Fileserver connections to any NSDB not in the fileserver's NSDB connection para…

I don't like (2) because it makes it harder to put everything back
when the compromised site gets cleaned up.

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 3 Jun 2013 17:12:01 -0400

> cert to use for this; should be trivial) for the example.net NSDB,

As an experiment, can you provide an example?

> when the compromised site gets cleaned up.

Agreed, though we may want such an operation for other reasons.

---

From: Nico Williams <[email-masked]>
Date: Mon, 3 Jun 2013 16:21:51 -0500

> As an experiment, can you provide an example?

Use openssl(1) to generate a key and a self-signed cert for it.  Throw
away the private key.  Use the cert.  The real NSDB a) won't have that
cert, b) won't have the corresponding private key.  (All this assuming
a decent source of entropy, of course.)

(There's lots of pages describing how to do this.)

> Agreed, though we may want such an operation for other reasons.

That's true enough.  Although junctions referring to the removed NSDB
presumably could remain dangling.  I'd rather mark the NSDB as
"removed, add no more junctions referring to it".  If junctions could
be enumerated that'd be even better (enumerated them, remove them,
then remove the NSDB from the fileserver's NSDB DB.

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 3 Jun 2013 17:50:59 -0400

> (There's lots of pages describing how to do this.)

Sorry, I was responding to your parenthetical.  By saying "we could publish," I assume you mean that such a bogus certificate would be published as part of a proposed standard.  I'm curious what something like that might look like.

> "removed, add no more junctions referring to it".

I assume that's something that FENCED could do, if we wanted it to.  I don't yet see the utility of having separate modes for "don't dereference this NSDB" and "don't use this NSDB for new junctions."

I believe it is currently a requirement for FEDFS_CREATE_JUNCTION to reject a request to create a junction using an NSDB whose connection parameters do not exist in the local NSDB connection parameters database.  Removing an NSDB from the local database would automatically have the effect of preventing the creation of junctions that refer to it.

> If junctions could

We may have decided a while back that we didn't want to place a requirement on implementations to enumerate junctions.  An implementation is free to enumerate if it is convenient.