Subject: [nfsv4] Fwd: New Version Notification for draft-haynes-nfsv4-flex-filesv2-00.txt

From: Thomas Haynes <[email-masked]>
Date: Tue, 8 Aug 2017 00:25:18 +0000

So here is a first hack at Flex Files v2. I thought it would come in much smaller than 11 pages,
so perhaps some pruning will be necessary.

I’m aware Ben had some comments on:

   Can I suggest putting in a typed opaque auth blob instead of explicit
   synthetic uid/gid fields?

I have purposely done:

   /// struct ffv2_data_server4 {
   ///     deviceid4               ffds_deviceid;
   ///     uint32_t                ffds_efficiency;
   ///     stateid4                ffds_stateid<>;
   ///     nfs_fh4                 ffds_fh_vers<>;
   ///     fattr4_owner            ffds_user;
   ///     fattr4_owner_group      ffds_group;
   ///     opaque_auth             ffds_auth;
   /// };

Because of AI1:

   [[AI1: after the lesson learned from ffds_stateid, we either need to put an
   array here or define all of the file handles to share the same
   credentials.  And as Olga points out in her email, this gets big
   fast.  Especially if we throw in many mirrored copies!  --TH]]

I.e., the ffds_auth is inside an array of ffv2_data_server4 which is
in itself inside an array of ffv2_mirror4 AND it could be argued that
ffds_auth (and ffds_user/ffds_group) need to be an array.

Even if we decide to make all FHs share the same credentials, we
eventually can hit a limit of the number of mirrored copies.

And Ben, the other reason not to use a typed opaque auth blob
is because the  ffds_user and ffds_group are strings, and do
not fit into AUTH_UNIX. We could define AUTH_SYNTHETIC. :-)

Begin forwarded message:

From: <[email-masked]<mailto:[email-masked]>>
Subject: New Version Notification for draft-haynes-nfsv4-flex-filesv2-00.txt
Date: August 7, 2017 at 5:11:45 PM PDT
To: Thomas Haynes <[email-masked]<mailto:[email-masked]>>

A new version of I-D, draft-haynes-nfsv4-flex-filesv2-00.txt
has been successfully submitted by Thomas Haynes and posted to the
IETF repository.

Name: draft-haynes-nfsv4-flex-filesv2
Revision: 00
Title: Parallel NFS (pNFS) Flexible File Layout v2
Document date: 2017-08-07
Group: Individual Submission
Pages: 11
URL:            https://www.ietf.org/internet-drafts/draft-haynes-nfsv4-flex-filesv2-00.txt
Status:         https://datatracker.ietf.org/doc/draft-haynes-nfsv4-flex-filesv2/
Htmlized:       https://tools.ietf.org/html/draft-haynes-nfsv4-flex-filesv2-00
Htmlized:       https://datatracker.ietf.org/doc/html/draft-haynes-nfsv4-flex-filesv2-00

Abstract:
  The Parallel Network File System (pNFS) allows a separation between
  the metadata (onto a metadata server) and data (onto a storage
  device) for a file.  The flexible file layout type is an extension to
  pNFS which allows the use of storage devices in a fashion such that
  they require only a quite limited degree of interaction with the
  metadata server, using already existing protocols.  This document
  describes two extensions to the flexible file layout type to allow
  for multiple stateids for tightly coupled NFSv4 models and an
  additional security mechanism for loosely coupled models.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org>.

The IETF Secretariat

---

From: David Noveck <[email-masked]>
Date: Tue, 8 Aug 2017 06:17:00 -0400

> not fit into AUTH_UNIX.

I don''t see how the fact that ffds_user and ffds_group, being strings,
do not fit into AUTH_UNIX can be argument for retaining them.  If
the existence of AUTH_UNIX is relevant to the matter, it would argue
for deleting ffds_user and ffds_group and subsuming them both
within ffds_auth.

The situation is the same regarding the fact there might be arrays of these
things.  Clearly, if one of these items needs to be an array, they all do..
If that's the case, it would be better to have one potentially troublesome
array rather than three.

On Mon, Aug 7, 2017 at 8:25 PM, Thomas Haynes <[email-masked]>
wrote:

> *From: *<[email-masked]>
> *Subject: **New Version Notification for
> draft-haynes-nfsv4-flex-filesv2-00.txt*
> *Date: *August 7, 2017 at 5:11:45 PM PDT
> *To: *Thomas Haynes <[email-masked]>
>
>
> until the htmlized version and diff are available at tools.ietf.org.
>
> The IETF Secretariat
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 8 Aug 2017 11:56:51 -0400

Hi Tom,

Are you planning to expand the 4.1.1? The same comment that I wrote on
my last email applies here where I suggested to specify 4things. It's
needed to have interoperable implementations. As something can choose
to return back TGTs (something like and AS_REP) and some might return
server tickets (something like a TGS-REP). Some server implementation
might encrypt needed info under the session key from the secured
communication, some might encrypt it with the user's main key. How
would the client know what to expect?

Also when I wrote about opaque_auth being 400bytes I was talking about
including just a single credential reply back. I'm looking on the
network trace and my AS_REP is 801bytes and TGS_REP is 960bytes. And
if somebody plans to include any authorization data in their tickets,
well that will blow up even more.

On Mon, Aug 7, 2017 at 8:25 PM, Thomas Haynes <[email-masked]> wrote:
> From: <[email-masked]>
> To: Thomas Haynes <[email-masked]>
>
>
> URL:
> https://www.ietf.org/internet-drafts/draft-haynes-nfsv4-flex-filesv2-00.txt
> Status:
> https://datatracker.ietf.org/doc/draft-haynes-nfsv4-flex-filesv2/
> Htmlized:
> https://tools.ietf.org/html/draft-haynes-nfsv4-flex-filesv2-00
> Htmlized:
> https://datatracker.ietf.org/doc/html/draft-haynes-nfsv4-flex-filesv2-00
>
>
> Abstract:

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 8 Aug 2017 14:54:58 -0400

Why pass Kerberos tickets around? Is there any reason not to just pass an
initialised RPCSEC_GSS session handle?

Cheers
  Trond

On 8 August 2017 at 11:56, Olga Kornievskaia <[email-masked]> wrote:

> >    [[AI1: after the lesson learned from ffds_stateid, we either need to
> put
> > an

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 8 Aug 2017 13:58:04 -0500

> initialised RPCSEC_GSS session handle?

There's not a standard serialization of the GSS security context object
that it contains, for transfer across the network.

-Ben

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 8 Aug 2017 15:19:42 -0400

For the client to authenticate to the server, it needs a service
ticket. A ticket carries a secret key encrypted with something known
only to the server. A client proves he knows the key inside the ticket
by encrypting a payload with that key (and presents the ticket as
well). KDC (MDS) encrypts that secret with something known only to KDC
and the client.

RPCSEC_GSS session handle is a number (allows to locate the real
crypto to use to decrypt the payload). It has no crypto, there is no
security there.

On Tue, Aug 8, 2017 at 2:54 PM, Trond Myklebust <[email-masked]> wrote:
>> > So here is a first hack at Flex Files v2. I thought it would come in
>> > much
>> > smaller than 11 pages,

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 8 Aug 2017 15:37:41 -0400

> that it contains, for transfer across the network.

I thought rfc1964 provides one, which is pretty much the basis for the user
library gss_krb5_lucid_context_v1_t typedef. Am I mistaken?

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 8 Aug 2017 14:49:16 -0500

> library gss_krb5_lucid_context_v1_t typedef. Am I mistaken?

I only see formats for security context establishment (negotiation) tokens,
which are exchanged so that the peers can establish a complete security
context -- I think you're mistaken.

AIUI the gss_krb5 lucid contexts came about due to the need to establish
a GSS security context in userspace and then export the keying material
into the kernel for per-message processing (for NFS).  I'm only about 95%
sure, but the information from a lucid context ought to be enough to
send from MDS to client and let the client use RPCSEC_GSSv3.  The
lucid context is a per-implementation extension, though, and we can't
use it in our standard without standardizing the lucid context first.

-Ben

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 8 Aug 2017 16:05:35 -0400

> library gss_krb5_lucid_context_v1_t typedef. Am I mistaken?

We have the case of chicken before the egg problem here.

Client has to send an AP_REQ to the server. He needs a ticket for
that. after the gss dance with the data server gss_krb5_lucid_context
is created.

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 8 Aug 2017 15:31:45 -0500

> that. after the gss dance with the data server gss_krb5_lucid_context

I think the proposal is that the MDS would send the AP_REQ, and generate a lucid
context to send to the client.  If the MDS also sends the RPCSEC handle,
the data server ought to be able to associate that with the same GSS security
context established by the MDS and handle RPCs from the client, even
though the peer's network address (as seen by the data server) has changed from
being the MDS to being the client.

-Ben

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 8 Aug 2017 16:51:59 -0400

> being the MDS to being the client.

Would an unmodified NFS server allow for this? Bruce might have better
idea. I don't think the server would accept "continued" gss sequence
on a new connection without going thru the gss dance.

What about the fact that it is possible that some NFS server might set
a rather short gss context lifetimes? The client would not be able to
renegotiate the new context to process with doing the IO. What happens
when the server reboots during the IO? It will lose the security
context and client is force to go to MDS every time?

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 8 Aug 2017 17:12:21 -0400

> use it in our standard without standardizing the lucid context first.

Right, but there are some issues with just passing back a KRB5 service
ticket too.

For one thing, how does the MDS choose which encryption scheme to use? It
knows what encryption the client is using for the RPCSEC_GSS session, but
presumably it has no idea what kind of ticket was used to negotiate that
session.

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 8 Aug 2017 17:27:58 -0400

> presumably it has no idea what kind of ticket was used to negotiate that

Since MDS = KDC then it would know which encryption scheme to use.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 8 Aug 2017 17:44:27 -0400

> Since MDS = KDC then it would know which encryption scheme to use.

The client will usually authenticate to the MDS using the corporate KDC.
The KDC on the MDS itself is not required to be the same.

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 8 Aug 2017 18:01:48 -0400

> On Aug 8, 2017, at 5:44 PM, Trond Myklebust <[email-masked]> wrote:
> 
> 
> 
> The client will usually authenticate to the MDS using the corporate KDC. The KDC on the MDS itself is not required to be the same.

But mds has access/copy of the principal database so it knows encryption types for the client server keys

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 8 Aug 2017 23:09:31 +0000

> On Aug 8, 2017, at 8:56 AM, Olga Kornievskaia <[email-masked]> wrote:
> 
> Hi Tom,
> 
> Are you planning to expand the 4.1.1?

I kept it as is in order to generate the discussion we are seeing.

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 8 Aug 2017 23:17:54 +0000

I don''t see how the fact that ffds_user and ffds_group, being strings,
do not fit into AUTH_UNIX can be argument for retaining them.

My argument would be the minimal set of changes necessary to make a ffv2 implementation.

One thing about the synthetic ids are that they are used to fence as much as they are used
as credentials. This may mean that we need both the ids and the auth field.

If

After having some time to think about this, I believe I have difference between these and the other
fields. Both the stateid and file handles are generated by the storage device - with loose coupling,
we cannot do anything to modify these. However, the synthetic ids, the auth token, and the
mode bits (which are never discussed but which play an important role), can be set by
the metadata server. So we can mandate that they are all the same across each instance
of the data file. If they are not, then the client will inform the metadata server in a LAYOUT_ERROR
(or LAYOUT_RETURN) that it cannot access the data and then the metadata server
can repair the access (mode/ids/auth token).

The auth token is not directly modifiable, but as the storage device validates it to the private auth server,
this should be sufficient.

On Mon, Aug 7, 2017 at 8:25 PM, Thomas Haynes <[email-masked]<mailto:[email-masked]>> wrote:
So here is a first hack at Flex Files v2. I thought it would come in much smaller than 11 pages,
so perhaps some pruning will be necessary.

I have purposely done:

Because of AI1:

Begin forwarded message:

Abstract:

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org<http://tools.ietf.org/>.

The IETF Secretariat

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 8 Aug 2017 18:27:30 -0500

> But mds has access/copy of the principal database so it knows encryption types for the client server keys 

There is a possible design where the MDS has access to the KDC's database, yes.
There are a couple other possible designs, though, and I don't think we've
ruled them out yet (whether due to technical constraints that cause them
to not actually work, policy decisions, or other).

(1) MDS is colocated with KDC or otherwise has access to the KDC database.

It's easy to get policy consistency between the KDC and MDS, find out about
supported enctypes, etc.  Issuing tickets is also easy and can be done with
direct database access and reuse a lot of code; the normal policy checks
are applied, authorization data can be included as usual, etc.  Actual principal
entries can be created/destroyed on demand for handing out as synthetic
credentials.  Access control can be done via synthetic client principal
name or authorization data in the tickets.

But, it adds more code running on the highly-privileged KDC machine, and a bug
in the MDS risks compromising the entire Kerberos realm -- a monstrous
rekeying/re-enrolling task.

(2) MDS is permitted by the KDC to use S4U2Proxy

This has some nice properties, like letting the KDC keep an audit trail
from the initial client authentication to the issued synthetic ticket and
separating the MDS from the KDC.  But, it is harder to create/destroy
client principal entries on demand, and most likely access control would
need to be done via authorization data in the resulting ticket.  (The KDC
might need to know to preserve and/or validate that authorization data
as part of the S4U2Proxy flow, but generally it should allow unknown
authorization data in the request to be preserved in the resulting ticket.)
The data server might need modification to use the Kerberos authorization
data for access control decisions.

(3) MDS knows data server's private key

This essentially makes the MDS trusted to emulate/hijack the data server's
NFS role, by making the key formerly shared between just the data server
and the KDC also shared by the MDS.  That key is used for authentication
and integrity (source) validation, so the MDS could impersonate the KDC
to the data server or vice versa.

This is very flexible about ticket "issuance", again allowing arbitrary
synthetic client principal names to be used and authorization data to be
inserted.  But, it causes the KDC to lose visibility into what is happening
in the Kerberos protocol and may have weaker auditability properties.

It also requires sharing the private key of the data server with another
party, which is in general something that should be avoided when using
Kerberos -- this sort of problem is one of the things that S4U2Proxy was
designed to address.

All three of these are only considering how to get a Kerberos ticket that
the MDS and/or client can ues to authenticate to the data server using
GSS-API; the details of how credentials and/or keys are conveyed from the
MDS to the client are a different problem, one that seems independent
of which of the above three options is taken.

-Ben

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 8 Aug 2017 19:55:46 -0400

On 8 August 2017 at 19:27, Benjamin Kaduk <[email-masked]> wrote:

> > > The client will usually authenticate to the MDS using the corporate
> KDC. The KDC on the MDS itself is not required to be the same.
> >
> >
There is a fourth option.

Give the MDS its own KDC and own domain that is completely separate and
untrusted by the corporate domain. The DSes would be required to recognise
principals from that domain, and not from the corporate domain.

The reason why that might be compelling is to ensure that principals that
are generated to support synthetic uids and gids may not be used to
authenticate to anything else in the corporate domain.

It also ensures that someone with a principal from the corporate domain has
no special privileges to access data on the DSes unless granted a layout by
the MDS.

IOW: the idea is to keep the authentication domain for the pNFS data path
entirely private to the storage system, while using the corporate domain to
authenticate to the MDS.

---

From: Olga Kornievskaia <[email-masked]>
Date: Wed, 9 Aug 2017 14:36:44 -0400

> entirely private to the storage system, while using the corporate domain to

I keep coming back to the question of what's different about assuming
that MDS and storage server belong to the same Kerberos realm (whether
or not it's corporate or private) and the tightly-coupled model where
everybody belongs to the same realms and normal ACL-based control is
used? Why is using normal principals and ACLs is ok in the
tightly-coupled model but it's not for loosely-coupled?

I don't know what's worse(better): private realm for storage (I think
is an over kill) or having MDS live on the same machine as the KDC.

The options I see for how to send the ticket back,  if it's a private
realm, then krb5p must be used and encrypt the secret that's also in
the service ticket. If it's a corporate realm, then krb5i could be
used and secret encrypted with the user's key. Maybe you can create a
return structure such that allows for either of the approaches. If
this flag is set, then client should decrypt with one and with another
otherwise. Then the implementors/administrators can decide whether
they want private realms or shared MDS/KDC appoarch.

Ben I don't see how S4UProxy fits. Isn't it typically
Alice->proxy->KDC and proxy as Alice->service. If this is the idea
where MDS will talk to the NFS server as Alice and then connection
trickery happen where Alice picks up the "same" connection without
ever doing proper AP_REQ to the storage server?

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 9 Aug 2017 15:17:37 -0400

On 9 August 2017 at 14:36, Olga Kornievskaia <[email-masked]>
wrote:

> >> > But mds has access/copy of the principal database so it knows
> encryption
> >> > types for the client server keys
> >>
> >> There is a possible design where the MDS has access to the KDC's
> database,
> >> yes.
> >> But, it adds more code running on the highly-privileged KDC machine,
> and a
> >> bug

In the tightly coupled model, the server has a special control channel to
the DS, and which it uses to enforce the layout semantics.

In the loosely coupled model, the control channel is NFSv3 and so layout
enforcement needs to be done by other means; in practice the only way to do
so is to use access controls. Please see the discussion about synthetic
uids and gids in the existing flex files draft for how it works.

> they want private realms or shared MDS/KDC appoarch.
The expectation is that the client would need to use krb5p to protect the
LAYOUTGET call to the MDS.

An alternative that might be worth exploring is to have the LAYOUTGET
return a stateid representing the credential to use, and then add a
separate LAYOUTAUTH call to convert the stateid into something that could
be used to set up the RPCSEC_GSS session to the DS.
The reason why that might be interesting is that it could allow the client
to reuse an existing RPCSEC_GSS session if one is already available.
It might also allow the client to share information such as supported
encryption types, etc as extra arguments to the LAYOUTAUTH call.

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 16 Aug 2017 21:21:50 +0000

On Aug 9, 2017, at 12:17 PM, Trond Myklebust <[email-masked]<mailto:[email-masked]>> wrote:

An alternative that might be worth exploring is to have the LAYOUTGET return a stateid representing the credential to use, and then add a separate LAYOUTAUTH call to convert the stateid into something that could be used to set up the RPCSEC_GSS session to the DS.
The reason why that might be interesting is that it could allow the client to reuse an existing RPCSEC_GSS session if one is already available.
It might also allow the client to share information such as supported encryption types, etc as extra arguments to the LAYOUTAUTH call.

I had also been thinking about a LAYOUTAUTH type of procedure. I didn’t want to build a handshake into
RPC-application-defined structured privilege assertion in RPCSEC_GSSv3.

As I understand RFC8178, this would fall into Section 6. The client would have to present LAYOUTAUTH to the mds and hence Section 6.

I’d like clarity on that before proposing a new op. :-)

---

From: David Noveck <[email-masked]>
Date: Wed, 16 Aug 2017 17:43:15 -0400

> As I understand RFC8178, this would fall into Section 6. The client would
have to present LAYOUTAUTH > to the mds and hence Section 6.

That's right.

I don't think RFC 8178 specifically addresses the matter but my expectation
was always that extensions  to the base protocol would be done in teir own
standards-track documents.  In this case, your layout specification would
normatively reference the extension.

On Wed, Aug 16, 2017 at 5:21 PM, Thomas Haynes <[email-masked]>
wrote:

>
> On Aug 9, 2017, at 12:17 PM, Trond Myklebust <[email-masked]> wrote:
>
> As I understand RFC8178, this would fall into Section 6. The client would
> have to present LAYOUTAUTH to the mds and hence Section 6.
>

---

From: [email-masked] (J. Bruce Fields)
Date: Mon, 28 Aug 2017 13:58:05 -0400

> on a new connection without going thru the gss dance.

The Linux server's gss processing is done in ignorance of any protocol
below that layer, and I can't think of a reason why we wouldn't accept a
gss context from a different IP address.

The split between kernel and rpc.gssd means that the Linux client always
does context initialization over a different connection from normal data
exchanges, so we can be pretty sure servers deal with that at least.

But I guess that wouldn't prevent other servers from binding the context
to a particular connection or IP address after that.  I don't think the
specs say anything either way.

--b.

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 28 Aug 2017 14:45:18 -0500

> exchanges, so we can be pretty sure servers deal with that at least.

Thanks for the details.

> specs say anything either way.

It's really something that would be specified at the (GSS) mechanism
layer, I think.  And Kerberos has generally moved away from address
restrictions since they fail in the presence of (e.g., residential) NAT.

-Ben