Subject: [nfsv4] NFSv4 on TLS

From: Chuck Lever <[email-masked]>
Date: Mon, 13 Aug 2018 12:00:30 -0400

https://www.linuxjournal.com/content/encrypting-nfsv4-stunnel-tls

There are some opinions about whether new versions of NFS should
REQUIRE a transport that guarantees confidentiality.

--
Chuck Lever

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 13 Aug 2018 12:49:26 -0500

> REQUIRE a transport that guarantees confidentiality.

Perhaps my response is too predictable, but "why not make it easier to use
Kerberos?"  If you're going to be setting up custom certs for an stunnel
TLS connection, you could just as well fake up a realm name, drop a
similarly fake keytab on the server, and vend a non-expiring service ticket
to the NFS client (encrypted to that same fake keytab).  A real keytab
could also potentially be used in this way, if the server desires.

-Ben

---

From: Chuck Lever <[email-masked]>
Date: Mon, 13 Aug 2018 14:32:28 -0400

> On Aug 13, 2018, at 1:49 PM, Benjamin Kaduk <[email-masked]> wrote:
> 

A real keytab would be significantly simpler to deploy than stunnel.
However, the author's performance comparison would show krb5p will
exact a significant performance impact.

Transport layer confidentiality is much easier to offload than
Kerberos is.

--
Chuck Lever

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 13 Aug 2018 15:04:04 -0400

> Transport layer confidentiality is much easier to offload than
I thought this kind of offloading was supposed to be enabled by the
RPCSEC_GSSv3 channel bindings as described in RFC 7861. Isn't that intended
to allow you to set up any suitably offloaded secure channel, and then hand
it over to the RPC client/server?

Cheers
  Trond

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Mon, 13 Aug 2018 21:33:02 +0200 (CEST)

There are two reasons why we use kerberos - transport level security and 'global' host independent
user identity. The first one is of course much simpler with TLS. With kerberos it's not that easy
to achieve. You need a dedicated service, client config and quite some knowledge.
Much, much simpler with certificates. Lot of tooling around it - openssl, tiny-ca... That's the
reason, why we use certificates for grid computing. But what about user identity? Well, with stunnel,
you can't solve it. You still need use (and synchronize) numeric uid and gid. Can we provide
consistent mapping and file's owership as it can be done with Krb+LDAP? That's why we can't
use NFS (or anything that relays on local uid+gid) for grid computing.

Unless, we do x509 certificates a first-class citizen for RPC,
using TLS is just a partial solution. However, sufficient for majority of the usecases.

Tigran.

----- Original Message -----
> From: "Chuck Lever" <[email-masked]>
> To: "Benjamin Kaduk" <[email-masked]>
> Cc: "NFSv4" <[email-masked]>
> Sent: Monday, August 13, 2018 8:32:28 PM
> Subject: Re: [nfsv4] NFSv4 on TLS

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 13 Aug 2018 20:34:42 -0500

> to achieve. You need a dedicated service, client config and quite some knowledge.

My point from the initial message is that if just the transport security is
desired (and not the identity information), the need for a dedicated
service and a lot of the extra knowledge could be removed.  That is, a tool
could be written that is essentially "enroll my NFS client to talk to this
NFS server and use krb5p for transport security", even in the absence of a
"real" Kerberos ecosystem.  The protocol-level pieces are all in place; all
that's left is someone with the need to write the code.

I'm also somewhat curious about the performance comparison for krb5p vs.
TLS (did I miss it in the linked article?), as we do have some discussion
about specifying AEAD kerberos enctypes for this sort of usage, but it
hasn't moved very fast yet.

-Ben

---

From: Chuck Lever <[email-masked]>
Date: Tue, 14 Aug 2018 11:05:39 -0400

> On Aug 13, 2018, at 9:34 PM, Benjamin Kaduk <[email-masked]> wrote:
> 

Near the end of the article there is a section entitled "Performance
Benchmarks" where the performance of NFSv4 on stunnel is compared with
fuse-sshfs. The author reports that fuse-sshfs performs better than
NFSv4.2 even without confidentiality, but fuse-sshfs does not provide
a rich set of filesystem features as NFSv4.2 does (especially around
ID mapping).

The krb5p implementation on Linux, for instance, has two well-known
performance issues:

- Encryption of payloads is expensive in terms of CPU cost and memory
bandwidth on both the client and server. I'm not aware of any offloaded
implementation of krb5p, but there are well-known examples of offloaded
transport layer confidentiality (eg. IPsec, RFC 6071).

- The GSS sequence number can serialize operations. If the GSS sequence
number window is overrun, spurious connection loss is triggered.

krb5p is not entirely confidential. All of the RPC consumer's message is
encrypted, but a portion of the RPC header is transmitted in the clear.

IMHO opinion it could be a valuable addition to RPC transports to start
integrating the IETF's ongoing generic TLS work so that a simple,
efficient, and standard (even commoditized) form of confidentiality can
be made easily available without the overhead of the RPCSEC framework.
</handwaving>

--
Chuck Lever
[email-masked]

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Tue, 14 Aug 2018 23:35:45 +0200 (CEST)

Hi Chuck, Ben,

probably decoupling security layer from identity information is not a bad idea.
At least for those who needs only data integrity and protection. And I agree,
that TLS is widely used, thus good optimized by many people. Mot of the penalty
is the initial handshake, which is ok for NFS.

Tigran.

----- Original Message -----
> From: "Chuck Lever" <[email-masked]>
> To: "Benjamin Kaduk" <[email-masked]>
> Cc: "Tigran Mkrtchyan" <[email-masked]>, "NFSv4" <[email-masked]>
> Sent: Tuesday, August 14, 2018 5:05:39 PM

---

From: Chuck Lever <[email-masked]>
Date: Thu, 23 Aug 2018 13:49:12 -0400

> On Aug 14, 2018, at 5:35 PM, Mkrtchyan, Tigran <[email-masked]> wrote:
> 
> Hi Chuck, Ben,
> 

For example:

https://tools.ietf.org/html/draft-ietf-mls-architecture-00

The architectural trend appears to be towards separating
authentication and transport security.

--
Chuck Lever
[email-masked]

---

From: Benjamin Kaduk <[email-masked]>
Date: Sun, 26 Aug 2018 14:48:56 -0500

> authentication and transport security.

(For background, I'm the responsible AD for the MLS WG.)  I don't think
MLS's discussions about how to apply per-message protection are as cut and
dried as that architecture document might make it seem, but you're right
that there are pressures from the IETF security community to reuse
well-established building blocks for transport security.  With my own
personal background I would also call GSS-API wrap tokens "well-established
building blocks", but I do acknowledge that they leave more metadata
unencrypted than TLS would.

-Ben

---

From: "Black, David" <[email-masked]>
Date: Sun, 26 Aug 2018 20:17:58 +0000

Chiming in from the cheap seats ...

> With my own

... I know that GSS-API is widely used with Kerberos authentication - is it widely used with other authentication mechanisms?  Please note that this is a deployment question, not a functionality (does it work?) question.

Thanks, --David

> -----Original Message-----
> From: nfsv4 [mailto:[email-masked]] On Behalf Of Benjamin Kaduk
> Sent: Sunday, August 26, 2018 3:49 PM
> To: Chuck Lever
> Cc: NFSv4

---

From: Benjamin Kaduk <[email-masked]>
Date: Sun, 26 Aug 2018 15:28:13 -0500

> ... I know that GSS-API is widely used with Kerberos authentication - is it widely used with other authentication mechanisms?  Please note that this is a deployment question, not a functionality (d…

The big GSS ("internet scale") mechanisms are basically Kerberos, the
SPNEGO negotiation pseudo-mechanism, and NTLM, from what I can tell.  (I
don't actually have great visibility into the NTLM deployments, and of
course one could argue whether it still counts as "GSS" when used with
Microsoft's SSPI directly.)  There are other mechanisms that have
substantial deployments -- the GSI mechanism
(http://grid.ncsa.illinois.edu/ssh/) that uses X.509 certificates is one,
and I get the impression from Martin Rex at SAP that they have a number of
customers using custom GSS mechanisms.  Some of those sites would be large
sites, but what with the mechanisms being custom, I don't expect them to be
used at a much broader scale than per-enterprise.

-Ben

---

From: Nico Williams <[email-masked]>
Date: Sun, 26 Aug 2018 16:47:26 -0500

> REQUIRE a transport that guarantees confidentiality.

So, the goal for a long time has been to use channel binding onto some
channel.  Ideally it would have been IPsec, but that requires RFC5660 to
get implemented, and that hasn't happened.

It is definitely possible to (and some do) run NFS over SSHv2, but TLS
would definitely be better (it's simpler and doesn't have flow control
that NFS does not need).

We could run NFS over TLS and then use GSS (with CB where possible) for
authenticating users, except that for single-user connections, if TLS
authenticates the one user, then GSS should not be further needed.

While we're here, it'd be nice if we standardized and implemented
something like the Globus (GSI) SSL GSS mechanism, which would look
like:

 - assign an OID for TLS, and one for DTLS (or just one OID and use a
   req_flag to indicate desire for TLS vs. DTLS)

 - add a mechanism attribute by which to indicate self-framing
   functionality

 - add a gss_get_token_length() function that takes a (possibly
   incomplete) token and outputs how many more bytes are needed / the
   token's length (when enough of the token is present)

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 27 Aug 2018 11:03:39 -0400

> On Aug 26, 2018, at 5:47 PM, Nico Williams <[email-masked]> wrote:
> 

> would definitely be better (it's simpler and doesn't have flow control

It is probably important to understand why some deployments prefer
to run NFS over a tunnel rather than use krb5p. See the Linux
Journal article that I posted at the start of this thread. I've
also heard requests from cloud operators who want to continue to
use AUTH_SYS but enforce the use of in-transit encryption.

> authenticates the one user, then GSS should not be further needed.

TLS is what I've been considering recently, but IMO the user
authentication mechanism should be a choice. krb5 user principals or
AUTH_SYS are both interesting.

Practical considerations for broad adoption:

- The encryption mechanism should be a "well-established building
block" that is likely to be available in a software implementation
provided by the OS, or in a hardware implementation on a commodity
NIC.

- The administrative cost is low. GSS Kerberos is not more widely
used because of the overhead of key distribution. Instead, a mech-
anism that uses certificates might be preferable because, like https,
a set of default globally useful certs can be populated on clients
in advance (via a package manager, for instance).

These can be augmented with additional certs that provide a relation-
ship to a local trust authority, allowing strong authentication of
endpoints.

- There is a reasonable performance cost. Tunneling and transport
encryption always seem to perform much better than krb5p. This is
especially critical for virtualized clients, which are already
performance-constrained.

- The mechanisms of transport confidentiality and user authentication
can be made separate. krb5p mandates the use of Kerberos user
credentials.

- In-transit encryption of any kind hobbles one of the most powerful
troubleshooting tools we have: network capture. Other tools will
have to be made available.

>   token's length (when enough of the token is present)

I'm not familiar with GSI. What are typical deployment steps, as
opposed to GSS Kerberos?

--
Chuck Lever

---

From: Nico Williams <[email-masked]>
Date: Mon, 27 Aug 2018 13:53:26 -0500

> use AUTH_SYS but enforce the use of in-transit encryption.

AUTH_SYS could be part of it -- it's supposed to be deprecated, but...

Another reason to do it is to reduce the amount of plaintext available
for traffic analysis, though in NFSv4 w/ GSS w/ confidentiality
protection there isn't that much plaintext available.

> AUTH_SYS are both interesting.

AUTH_SYS with UIDs/GIDs is too limited.  I have no objection to a
mechanism like AUTH_SYS-over-TLS that uses name@domain to be consistent
with the rest of NFSv4.

> [...]

Kerberos is not more widely deployed because its cross-realm system
doesn't scale to Internet scale at this time.  The key orchestration
problems are solved (e.g., in Windows it's automatic, and there are
several open source systems for orchestration in Heimdal or MIT
environments, and Red Hat has its own solution too now).

> especially critical for virtualized clients, which are already

Yes: because there are fewer cryptographic contexts lying around to put
pressure on caches.  This was the rationale for channel binding to
IPsec, which RFC5660 was the meant to be the first half of.

> can be made separate. krb5p mandates the use of Kerberos user

It's still GSS, which is pluggable.  It'd doesn't have to be Kerberos.
(krb5p has to be Kerberos, yes, but you can have other GSS mechs.)

> have to be made available.

Wireshark can handle this, you just have to give it access to the
server's keys.

If anything, this will be harder to do with TLS because in 1.3 you must
always have PFS, and so having access to the server's private keys won't
help wireshark.  Instead you have to have the ability to dump plaintext
network traffic on the server (or client) itself.

> I'm not familiar with GSI. What are typical deployment steps, as

It's just a GSS interface to SSL (TLS).  It's just SSL (TLS).

Nico
--

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Tue, 28 Aug 2018 10:54:22 +0200 (CEST)

----- Original Message -----
> To: "Nico Williams" <[email-masked]>
> Sent: Monday, August 27, 2018 5:03:39 PM

> I'm not familiar with GSI. What are typical deployment steps, as

GSI is certificate based with some extensions. it has SSL and GSS implementations.
However, it requires it's own ecosystem and tooling. After almost 20 years of using
it for High Energy Physics (CERN experiments and others) we are moving away from in
favor of pure TLS and auth tokens.

Tigran.

---

From: David Noveck <[email-masked]>
Date: Tue, 28 Aug 2018 07:30:40 -0400

> AUTH_SYS could be part of it

Yes, but the question we need to address is whether it should be part of it.

> it's supposed to be deprecated, but...

With the publication of RFC3530 it was clearly deprecated.  That happened
over
fifteen years ago, and while use of AUTH_SYS has decreased, it does not seem
that use of AUTH_SYS is going to be gone in the near future.

The security of AUTH_SYS used in the clear is pretty much non-existent and
the
deprecation of this deployment  model in existing NFSv4 RFCs is, if
anything, too mild.
The case of AUTH_SYS together with transport-level encryption is an
interesting
one which is not really addressed in existing NFSv4 RFCs.   I think we need
to
consider how we will address that situation.

> AUTH_SYS with UIDs/GIDs is too limited.

As far as I know, AUTH_SYS includes 32-bit UIDs/GIDs.   Those seemed ample
when
AUTH_SYS was designed.

It certainly is too limited now, but it is difficult to change. Even more
of a
barrier than the existiing OTW formats, is the fact that most flle systems
used with NFS reserve 32 bits in the inode for each of owner and group.
Further complicating matters is that in-kernel VFS's and POSIX API's are
built around this very limited approach to user identity.

> I have no objection to a mechanism like AUTH_SYS-over-TLS

I don't know of any AUTH_SYS mechanism that uses anything other than
32-bit ids.

> that uses name@domain to be consistent with the rest of NFSv4.

Within NFSv4 proper, you could use name@domain, but there is really no
point if you are using AUTH_SYS.    You are allowed to use numeric ids
as attributes and most people using AUTH_SYS choose to do so, to avoid
the overhead of a translation that provides no benefits.

---

From: Rick Macklem <[email-masked]>
Date: Tue, 28 Aug 2018 11:57:15 +0000

David Noveck wrote:
[stuff snipped]
>As far as I know, AUTH_SYS includes 32-bit UIDs/GIDs.   Those seemed ample when
>AUTH_SYS was designed.
>
>It certainly is too limited now, but it is difficult to change. Even more of a
I think he was referring to a new authentication flavour that had the
"user@domain" string as the authenticator. (There is a 400byte limit, but
that might be sufficient.) I might have called it AUTH_NAME or AUTH_OWNER.

I'd guess that 99.9% of FreeBSD NFS usage is AUTH_SYS, although Kerberos is
in the system. I do think a way to use TLS (or similar) to encrypt at the transport
level would be a good thing and would get used more widely.

Many of the FreeBSD sites (I won't guess the %) use AUTH_SYS but ignore the
gid list and generate a gid list on the server from the uid. As such, I don't think
putting gids/owner_group names in a new authentication mechanism is needed.

For folks that have a 32bit uniform uid space, AUTH_SYS in transport encryption
seems ok to me.
For sites where the uid space isn't uniform or doesn't fit in 32bits, a "user@domaim"
to uid/gid list mapping in the server can be done using this new one.

>the overhead of a translation that provides no benefits.
They could use this mapping with AUTH_OWNER when AUTH_SYS won't
work for them.

rick
ps: I, for one, am not interested in implementing a new GSSAPI mechanism that
      isn't in the open source Kerberos distros. As such, any complicated setup
      involving writing of a lot of new code probably wouldn't make it in FreeBSD
      for a long time. However, something that mostly uses the extant code for
      supporting "https" should be doable.
[more stuff snipped]