Subject: [nfsv4] Secdir last call review of draft-ietf-nfsv4-flex-files-15

From: Brian Weis <[email-masked]>
Date: Fri, 12 Jan 2018 10:45:42 -0800

Reviewer: Brian Weis
Review result: Has Issues

I have reviewed this document as part of the security directorate's ongoing
effort to review all IETF documents being processed by the IESG. These comments
were written primarily for the benefit of the security area directors. Document
editors and WG chairs should treat these comments just like any other last call
comments.

This document defines the Parallel Network File System (pNFS), which allows a
storage system to store the metadata for files onto a metadata server, and data
of that file onto a separate storage device. The document focuses on the
interaction between the two types of servers, and the requirements of clients
using a pNFS system.

The document describes two models by which the metadata server and file storage
server can communicate. One is a “tight coupling model”, in which the two types
of servers directly communicate. In this model the metadata server can ensure
that file access control state is synchronized between the two servers, and I
believe that level of access control is equivalent to NFS v4.x today. From a
security considerations perspective, this is perhaps best summarized in Section
2.2:

  “With tightly coupled storage devices, the metadata server sets the
   user and group owners, mode bits, and ACL of the data file to be the
   same as the metadata file.  And the client must authenticate with the
   storage device and go through the same authorization process it would
   go through via the metadata server.”

The other model is a “loose coupling model”, in which the client is
authenticated and authorized by the metadata server, and is given access to the
file server through the use credentials. However, there is no direct
communications between the metadata server and the file server. The metadata
server does not seem to provide a fine grained access control to files on the
file server, but instead relies upon the client to enforce a less granular
level of access control.

The Security Considerations section is well-written and helpful. I do have some
suggestions for improving it,  and also some questions.

(1) Section 2.2: should the “recommended” in the Note be RECOMMENDED?
Restricting access control to just superuser have access to the storage device
seems like a valuable advice.

(2) Section 2.2: The NOTE probably should provide guidance regarding protecting
the network connections between the metadata server and the file server. For
example, something like: “Communications between the metadata server and file
server need to secured. If there is not adequate physical security (i.e., the
servers are not co-located in a physically secure area), their communications
MUST be protected for confidentiality (e.g., using TLS or IPsec).”

(3) From the description of synthetic uids/gids in Section 2.2, I believe that
the full range of file permissions (e.g., owner r/w/x)  is honored in the
tightly coupled model, but are restricted in the loosely coupled model to
“owner r/w”, “group r”. If that is correct, I think the security considerations
should explicitly mention that the effect of access permissions is reduced on
the file server files.

(4) Section 2.2. In this phrase “it may be assigned some random synthetic
uid/gid to deny access”, would a better word for “random” be “unused” or
“unpredictable”? Also what is the security value intended by the use of a
“random” value for synthetic uids/gids?

(5) In the “loose coupling model” file access actually enforced by the client,
which presumably can be manipulated by an evil user. This is clearly stated n
Section 2.2: “ The client is thus solely responsible for enforcing file
permissions in a loosely coupled model.”  Also, Section 2.2.1 mentions “a
SETATTR would be sent to the storage device to set the owner and group of the
data file”, and since here is no control protocol between the metadata server
and the file server so I assume it is sent by the client.

Put another way, it seems as if any client that has credentials allowing it to
access to the file server can access any file on the file system if it is
chooses to do so (e.g.., if it is a malicious client). Is that correct?

Unless I’m mistaken, this is a serious reduction of actual security from NFS
4.x, and also from the tight coupling model. This should be clearly stated in
the security considerations section, so that protocol adopters and users
understand the additional risks of the loose coupling model. For this  reason,
I've marked the review with "Has Issues" rather than "Has Nits".

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 12 Jan 2018 19:41:22 +0000

Hi Brian,

Thanks for the review - comments inline..

> On Jan 12, 2018, at 10:45 AM, Brian Weis <[email-masked]> wrote:
> 
> Reviewer: Brian Weis

[1] No, there is still direct communication between the metadata server and the file server.

The difference is that it is via a storage protocol, in this case NFS.

The access control is enforced by the metadata server. The client only get access
to the file if it can get a layout. The layout can either provide read/write access
or only read access.

The metadata server can revoke that access at any point, so it is not
correct to say that it depends on the client to enforce a less granular level
of access control.

> seems like a valuable advice.

sure

> MUST be protected for confidentiality (e.g., using TLS or IPsec).”

No, the MUST is too strong here. We can recommend procedures, we can not REQUIRE them.

I am fine with:

Communications between the metadata server and file
server SHOULD be secured. If there is not adequate physical security (i.e., the
servers are not co-located in a physically secure area), their communications
SHOULD be protected for confidentiality (e.g., using TLS or IPsec).”

> should explicitly mention that the effect of access permissions is reduced on

[2] I think the confusion here is that the files on storage device are not
in the file hierarchy as seen on the metadata server.

On a file by file basis on the storage device, we are giving the client either
write modification access, read access, or no access at all. The client cannot
remove the file (it would need to know the parent directory FH and the
name of the file - neither of which it has).  It can effectively remove the content
if it has write permission (but nothing would prevent this in a tightly
coupled model). 

> “random” value for synthetic uids/gids?

In an earlier review (by Linda Dunbar), that phrase was changed:

  On the storage device, it may be assigned some random synthetic uid/
  gid to deny access:

  -rw-r-----    1 19452   28418    1697 Dec  4 11:31 data_ompha.c

  When the file is opened on a client and accessed, it will try to get a
  layout for the data file. Since the layout knows nothing
  about the user (and does not care), whether the user loghyr or
  garbo opens the file does not matter.  The client has to present
  an uid of 19452 to get write permission. If it presents any other
  value for the uid, then it must give a gid of 28418 to get read
  access.

  Further, if the metadata server decides to fence the file, it may
  change the uid and gid as such:

  -rw-r-----    1 19453   28419    1697 Dec  4 11:31 data_ompha.c

What we could suggest would be something along the lines of:

  On the storage device, it may be assigned some unpredictable
  synthetic uid/gid to deny access.
  …
  Further, if the metadata server decides to fence the file, it
  should change the uid and/or gid such that these values
  neither match earlier values for that file nor match
  a predictable change based on an earlier fencing.

Ugh, not sure I like the words.

uid[N+1] != uid[N-1] && uid[N+1]  != change(uid[N-1] -> uid[N]).

I.e., do not set to recent old values (might fence back to an uid
an older client already has) and do not use a fixed increment.

> and the file server so I assume it is sent by the client.

It is subtle, but this goes back to the fact (see [1]) that “no control protocol” does not
imply “no communication between the metadata server and the storage
device”.  I.e., the MDS does the SETATTR.

Also, I notice you are using the phrase “file server”, which probably impacts
your view in your point 3 (see my [2]) - the term we use is storage
device. There is no requirement that this device serve up data files
as if they were in a filesystem. The storage device is limited in the operations
that it can respond to from the client based on the information it is
presented with.

For example, the exports on the storage device can be configured such
that the metadata server is the only one which can mount the exports.
And since the metadata server only hands out File Handles to the
clients, they would not be able to walk the namespace of the storage
device.

> chooses to do so (e.g.., if it is a malicious client). Is that correct?

As with any NFS filesystem which uses an AUTH_SYS security model,
each of the file handle, uid, and gid can be forged to gain access.

In other words, this is nothing new.

The security model for NFSv4.2 (and based on that, NFSv4.0 and NFSv4.1):

[RFC7862]
17.  Security Considerations

   NFSv4.2 has all of the security concerns present in NFSv4.1 (see
   Section 21 of [RFC5661]), as well as those present in the server-side
   copy (see Section 4.9) and in Labeled NFS (see Section 9.6).

Which leads us to [RFC5661]:

2.2.1.1.1.2.  Security Mechanisms for NFSv4.1

   RPCSEC_GSS, via GSS-API, normalizes access to mechanisms that provide
   security services.  Therefore, NFSv4.1 clients and servers MUST
   support the Kerberos V5 security mechanism.

   The use of RPCSEC_GSS requires selection of mechanism, quality of
   protection (QOP), and service (authentication, integrity, privacy).
   For the mandated security mechanisms, NFSv4.1 specifies that a QOP of
   zero is used, leaving it up to the mechanism or the mechanism's
   configuration to map QOP zero to an appropriate level of protection.
   Each mandated mechanism specifies a minimum set of cryptographic
   algorithms for implementing integrity and privacy.  NFSv4.1 clients
   and servers MUST be implemented on operating environments that comply
   with the REQUIRED cryptographic algorithms of each REQUIRED
   mechanism.

2.2.1.1.1.2.1.  Kerberos V5

   The Kerberos V5 GSS-API mechanism as described in [5] MUST be
   implemented with the RPCSEC_GSS services as specified in the
   following table:

To paraphrase it, NFSv4.x clients and servers MUST support
the Kerberos V5 security mechanism, but installations are not
required to use Kerberos.

I.e., this:

mount -o vers=4.2,sec=sys server:/ /mnt

means that any malicious user can use any client which can connect
to the server to do whatever they want.

In other words, this is not a reduction of the actual security from NFSv4.x.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Brian Weis (bew)" <[email-masked]>
Date: Fri, 12 Jan 2018 23:54:25 +0000

Hi Thomas,

On Jan 12, 2018, at 11:41 AM, Thomas Haynes <[email-masked]<mailto:[email-masked]>> wrote:

Hi Brian,

On Jan 12, 2018, at 10:45 AM, Brian Weis <[email-masked]<mailto:[email-masked]>> wrote:

Reviewer: Brian Weis

Thanks for the clarification. To a non-NFS expert the relationship between layouts and
access control  wasn’t obvious. I wonder if this discussion could be added to Security
Considerations?

sure

I am fine with:

I understand the hesitation to make it a MUST, because it’s on a protocol
that isn’t described in this document. But I think it’s reasonable to describe
a risk and make a requirement that the risk be mitigated. Certainly it’s not
reasonable to require a particular method to be used to protect the protocol
(which is why I suggested “e.g., using TLS or IPsec” above rather than “i.e.”).

With today’s pervasive privacy threats to network traffic it’s hard to
justify allowing metadata to traverse unprotected networks as clear-text
(i..e, available to  passive eavesdroppers). It’s also hard or to justify allowing
security-relevent protocols to be transferring it without cryptographic integrity
protection. It seems important to remind protocol developers that both
concerns are applicable. How about something like this, where more rationale
is given. Note that the requirement that the protocol must be protected, but
no particular method of protection is not mandated.

"Communications between the metadata server and file
server MUST be secure from eavesdroppers and man-in-the-middle protocol
tampering. The security measure could be due to physical security (e.g., the
servers are co-located in a physically secure area), from encrypted communications,
or some other technique.”

Got it, thanks.

So maybe in the first paragraph you mean “… some unpredictable and
unused by any current client”?

Thanks for the explanation … makes sense.

[RFC7862]

I.e., this:

Thanks for the extensive explanation. It resolves the issue that I perceived.

Thanks,
Brian

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

--
Brian Weis
Security, CSG, Cisco Systems
Telephone: +1 408 526 4796
Email: [email-masked]<mailto:[email-masked]>

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 16 Jan 2018 18:56:15 +0000

On Jan 12, 2018, at 3:54 PM, Brian Weis (bew) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Thomas,

Hi Brian,

Reviewer: Brian Weis

Hi Brian,

I’ve already got that (and yes, I understand it is not sufficient if I have to spell it out :-):

15.  Security Considerations
...
   The metadata server is primarily responsible for securing the data
   path.  It has to authenticate the client access and provide
   appropriate credentials to the client to access data files on the
   storage device.  Finally, it is responsible for revoking access for a
   client to the storage device.

   The metadata server enforces the file access-control policy at
   LAYOUTGET time.  The client should use RPC authorization credentials
   for getting the layout for the requested iomode (READ or RW) and the
   server verifies the permissions and ACL for these credentials,
   possibly returning NFS4ERR_ACCESS if the client is not allowed the
   requested iomode.  If the LAYOUTGET operation succeeds the client
   receives, as part of the layout, a set of credentials allowing it I/O
   access to the specified data files corresponding to the requested
   iomode.  When the client acts on I/O operations on behalf of its
   local users, it MUST authenticate and authorize the user by issuing
   respective OPEN and ACCESS calls to the metadata server, similar to
   having NFSv4 data delegations.

   If access is allowed, the client uses the corresponding (READ or RW)
   credentials to perform the I/O operations at the data file's storage
   devices.  When the metadata server receives a request to change a
   file's permissions or ACL, it SHOULD recall all layouts for that file
   and then MUST fence off any clients still holding outstanding layouts
   for the respective files by implicitly invalidating the previously
   distributed credential on all data file comprising the file in
   question.  It is REQUIRED that this be done before committing to the
   new permissions and/or ACL.  By requesting new layouts, the clients
   will reauthorize access against the modified access control metadata.
   Recalling the layouts in this case is intended to prevent clients
   from getting an error on I/Os done after the client was fenced off.

How about:

   If access is allowed, the client uses the corresponding (READ or RW)
   credentials to perform the I/O operations at the data file's storage
   devices. …

becomes

  The combination of file handle, synthetic uid, and gid in the layout
  are the way that the metadata server enforces access control to the
  data server. The directory namespace on the storage device SHOULD
  only be accessible to the metadata server and not the clients. In that case,
  the client only has access to file handles of file objects and not
  directory objects. Thus, given a file handle in a layout, it is not
  possible to guess the parent directory file handle. Further, as the
  data file permissions only allow the given synthetic uid read/write
  permission and the given synthetic gid read permission, knowing
  the synthetic ids of one file does not necessarily allow access to any
  other data file on the storage device.

  The metadata server can also deny access at any time by fencing
  the data file, which means changing the synthetic ids. In turn, that
  forces the client to return its current layout and get a new layout if
  it wants to continue IO to the data file.

  If the configuration of the storage device is such that clients
  can access the directory namespace, then the access control
  degrades to that of a typical NFS server with exports with a
  security flavor of AUTH_SYS.  Any client which is allowed
  access can forge credentials to access any data file. The
  caveat is that the rogue client might have no knowledge of
  the data file’s type or position in the metadata directory
  namespace.

   If access is allowed, the client uses the corresponding (READ or RW)
   credentials to perform the I/O operations at the data file's storage
   devices. …

Or do you want me to pull that new text into a new subsection 15.1 and
slam the other subsections down?

sure

I am fine with:

I understand the hesitation to make it a MUST, because it’s on a protocol
that isn’t described in this document. But I think it’s reasonable to describe
a risk and make a requirement that the risk be mitigated.

I do not disagree with that statement.

Certainly it’s not

I’m still hesitant, in part because I can’t make an ironclad rule as a standard’s
author that I know I am going to violate with my own implementation.

Also, I know that there might be existent storage devices which do not support
NFS over TLS/IPsec/etc. Yes, physical security is the answer here, but I cannot
in full confidence expect that an actual deployment meets all of these criteria.
At some point, the burden is off of the protocol implementation and onto the
deployment.

If we change the MUST to a SHOULD, I am perfectly fine making the change. :-)

Got it, thanks.

[RFC7862]

I.e., this:

Thanks,
Brian

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 16 Jan 2018 18:59:35 +0000

On Jan 16, 2018, at 10:56 AM, Thomas Haynes <[email-masked]<mailto:[email-masked]>> wrote:

Hi Thomas,

Hi Brian,

Reviewer: Brian Weis

Hi Brian,

How about:

becomes

sure

I am fine with:

Certainly it’s not

By the way, none of the other layout types make this type of requirement on
their control protocols. The overall security of the data lies in the combination
of metadata server, storage device, and client.

Got it, thanks.

[RFC7862]

I.e., this:

Thanks,
Brian

---

From: "Brian Weis (bew)" <[email-masked]>
Date: Thu, 25 Jan 2018 18:44:48 +0000

Hi Thomas,

On Jan 16, 2018, at 7:56 PM, Thomas Haynes <[email-masked]<mailto:[email-masked]>> wrote:

Hi Thomas,

Hi Brian,

Reviewer: Brian Weis

Hi Brian,

How about:

becomes

Your text above sounds great to me, and adding it to security  considerations as you
suggest is fine.

sure

I am fine with:

Certainly it’s not

Hmmm. Well, I guess with the risks explicitly mentioned there’s less harm in a
SHOULD. If you’re OK with the rest of the wording, go ahead and substitute the
SHOULD for the MUST.

Thanks,
Brian

Got it, thanks.

[RFC7862]

I.e., this:

Thanks,
Brian

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 25 Jan 2018 23:50:48 +0000

On Jan 25, 2018, at 10:44 AM, Brian Weis (bew) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Thomas,

Thanks,
Brian

Hi Brian,

Thanks for the response - I had modified this slightly as per Kathleen’s review.
The new version is posted.

Thanks,
Tom