Subject: Re: [nfsv4] New wording of Security Section for Flex Files

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 1 Aug 2017 13:29:34 -0400

For some reason I didn't not receive the Tom's original email that
addressed the comments so I'm replying within Ben's reply.

On Thu, Jul 27, 2017 at 10:23 PM, Benjamin Kaduk <[email-masked]> wrote:
> On Tue, Jul 25, 2017 at 01:07:02AM +0000, Thomas Haynes wrote:
>> I hope the following addresses the issues brought up by Ben and Olga;

I'd like the say the new wording is more specific but I still think
the proposal tries to fit Kerberos security where it doesn't fit.

> I think it does, thanks.  A couple notes inline.
>
>> First paragraph of Section 15 is replaced by:
>>
>>    The pNFS extension partitions the NFSv4.1+ file system protocol into
>>    two parts, the control path and the data path (storage protocol).
>>    The control path contains all the new operations described by this
>>    extension; all existing NFSv4 security mechanisms and features apply
>>    to the control path (see Sections 1.7.1 and 2.2.1 of [RFC5661]).  The
>>    combination of components in a pNFS system is required to preserve
>>    the security properties of NFSv4.1+ with respect to an entity
>>    accessing data via a client, including security countermeasures to
>>    defend against threats that NFSv4.1+ provides defenses for in
>>    environments where these threats are considered significant.
>>
>> Replace Section 15.1 with:
>>
>>    RPCSEC_GSS version 3 (RPCSEC_GSSv3) [RFC7861] could be used to
>>    authorize the client to the storage device on behalf of the metadata
>>    server.  This would require that each of the metadata server, storage
>>    device, and client would have to implement RPCSEC_GSSv3.  The second
>>    requirement does not match the intent of the loosely coupled model
>>    that the storage device need not be modified.  (Note that this does
>>    not preclude the use of RPCSEC_GSSv3 in a loosely coupled model.)
>>
>>    Under this coupling model, the principal used to authenticate the
>>    metadata file is different than that used to authenticate the data
>>    file.  For the metadata server, the RPC credentials would be
>>    generated by the same source as the client.  For RPC credentials to
>
> "by the same source as" feels like a slightly awkward phrasing that leaves
> me wondering if I don't quite properly understand the intended sentiment.
> Isn't it basically required that the client has RPC credentials to talk
> to the metadata server in order to do anything?
>
>>    the data on the storage device, the metadata server would be
>>    responsible for their generation.  Such "credentials" SHOULD be
>>    limited to just the data file be accessed.  Using Kerberos V5 GSS-API
>>    [RFC4121], some possible approaches would be:
>>
>>    o  a dedicated/throwaway client principal name akin to the synthetic
>>       uid/gid schemes.

If the identity is a throwaway, I don't see how this would work in
practice (how would the user be informed of the passwords that go
together with the throwaway identity to be used by NFS)? It just
doesn't seem like a practical scheme to suggest.

>>    o  authorization data in the ticket.
>>    o  an out-of-band scheme between the client and metadata server.

Is this meant to be an "out-of-band security scheme" not necessary
Kerberos or GSS. That's something I would agree with. However, what's
worse suggesting that GSSv3 should be implemented by the model vs just
stating that some out of band scheme is used. Earlier in the spec, it
says that "With a loose coupling, the only control protocol might be a
version of NFS.  As such, semantics for managing security, state, and
locking models MUST be defined." Is stating that an out-of-band scheme
is used satisfies the "MUST" of semantics for managing security?

> Now I feel like I should have spent more time to carefully word the listing
> in my original email :)
> Though I think this text is fine to convey the needed information, at least.
>
>
>>    [[AI15: Editorial Note: I have *SHOULD* and not *MUST* because there
>>    might be some limit on the number of outstanding credentials due to
>>    either the number of files or the number of client.  Feel free to
>>    correct me.  --TH]]
>
> I can think of schemes where there are no limits, but I am not prepared
> to guarantee that all schemes one might want to use would be free of limits.
> So the SHOULD seems fine to me.
>
> Semi-relatedly, I forget exactly how much trust/coupling there is supposed
> to be between the data and metadata servers/operators in this scheme.
> In particular, if the metadata server is sufficiently trusted that it can
> have a copy of the data server's kerberos credentials (the nfs/ keytab),
> then there is quite a lot of flexibility, since the metadata server can
> literally construct an arbitrary ticket and encrypt it to "itself" (i.e.,
> the data server's keytab).  But I don't know whether it's appropriate
> to mention this example in the document or not.
>
>>    Depending on the implementation details, fencing would then be
>>    controlled either by expiring the credential or by modifying the
>
> Kerberos doesn't have a revocation (expiry prior to the original expiry)
> story at all.  But since we are talking about RPCSEC_GSS, this text
> is fine, since other GSS mechanisms might allow it :)
>
> -Ben
>
>>    synthetic uid or gid on the data file.  I.e., if the credentials are
>>    at a finer granularity than the synthetic ids, it might be possible
>>    to also fence just one client from the file.
>>
>> Replace 15.1.2 with:
>>
>>    With tight coupling, the principal used to access the metadata file
>>    is exactly the same as used to access the data file.  The storage
>>    device can use the control protocol to validate any RPC credentials.
>>    As a result there are no security issues related to using RPCSEC_GSS
>>    with a tightly coupled system.  For example, if Kerberos V5 GSS-API
>>    [RFC4121] is used as the security mechanism, then the storage device
>>    could use a control protocol to validate the RPC credentials to the
>>    metadata server.
>>

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 1 Aug 2017 18:18:49 +0000

> On Jul 27, 2017, at 7:23 PM, Benjamin Kaduk <[email-masked]> wrote:
> 

I think anything here will be “slightly awkward”.

For the metadata server, the RPC credentials are generated and
verified as in the non-pNFS case.

A little better, but still reads a bit off.

> to the metadata server in order to do anything?

Yes

> So the SHOULD seems fine to me.

Thanks, I’ll remove the AI15.

> to be between the data and metadata servers/operators in this scheme.

The trust is that for the exported part of the name space, the metadata server
has full super user rights.

Note that if the storage device is used for other purposes, the metadata
server should not have super user rights.

> to mention this example in the document or not.

This could be an out-of-band solution. 

I really want to balance providing enough context of how the system
can be secured against being too specific. I.e., have the hand waving
be plausible.

> is fine, since other GSS mechanisms might allow it :)

I think the lack of upper case use of RFC 2119 words covers it here.

---

From: David Noveck <[email-masked]>
Date: Tue, 1 Aug 2017 14:29:33 -0400

> Is stating that an out-of-band scheme

Probably not, but the "MUST" can always be changed to a "SHOULD" :-)

The problem is deeper here.   The job of a standards-track document that
defines a protocol or layout type is to clearly tell the implementer what
must be done to implement the protool or layout type.  Despite the possble
issues with the specificity of any of these three possible alternatives,
the big problem is the fact that there is more than one.  If the client and
 MDS do not pick the same one, or they do and that one is not fully
specfied, they will not nteroperate.  If that is the case, then the
document is not ready.

I think the treatment of security with tight coupling is sufficiently clear
to implement from and has been for a while.  With regard to security for
loose coupluing, what you have is essentially a promise to provide a
specfication at some later date.  I don't think the IESG is going to be OK
with that.

You have focusing on it being plausible in the sense of convincing people
that this can be done
(i.e. that a definition along these lines can be produced) which it can,
but I think the IESG is going to want you to show them that it has been
done (i.e. that the definiton has been arrived at), and you are a ways away
from being there.

On Tue, Aug 1, 2017 at 1:29 PM, Olga Kornievskaia <[email-masked]>
wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 1 Aug 2017 19:43:56 +0000

> On Aug 1, 2017, at 10:29 AM, Olga Kornievskaia <[email-masked]> wrote:
> 

I agree that the traditional model of trust does not apply here, that
we have to shoehorn in a level of trust that basically states the
mds owns the data on the storage device.

But I believe if if we are willing to accept that trust model, RPCSEC_GSS
can be used here.

> doesn't seem like a practical scheme to suggest.

I *think* the intent is that these identities would not have passwords.

> worse suggesting that GSSv3 should be implemented by the mode

“should” was not used, “could” was used. I may be guilty of trying to
describe an approach without having an actual implementation, but
I can envision if the same vendor supplied each of the metadata server,
client, and the storage device and did not want to define a control
protocol, they could use GSSv3’s RPC-application-defined
structured privilege assertion in a manner described in
Section 4.9.1 of RFC 7862.

I can be much more explicit in citing this approach, but I believe the
implementation constraints means that none will utilize it. However,
if multiple implementors wanted to do this, we could provide an
additional standards track document detailing the approach.

> l vs just

No, I do not think so.

Even if NFSv4 were just a data center implementation, we would still
be asked (by the IETF and users/customers) to supply security.

My goal in this document is to provide enough plausible approaches
without handcuffing an implementation. I.e., if I supplied XDR and
verbiage for equivalent structures to copy_from_auth_priv,
copy_to_auth, and copy_to_auth_priv and with no intent to
ever implement it, then an implementation with 3rd party storage
devices would be prohibited.

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 1 Aug 2017 19:59:02 +0000

The problem is deeper here.   The job of a standards-track document that defines a protocol or layout type is to clearly tell the implementer what must be done to implement the protool or layout type.  Despite the possble issues with the specificity of any of these three possible alternatives, the big problem is the fact that there is more than one.  If the client and  MDS do not pick the same one, or they do and that one is not fully specfied, they will not nteroperate.  If that is the case, then the document is not ready[https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif].

I think the treatment of security with tight coupling is sufficiently clear to implement from and has been for a while.  With regard to security for loose coupluing, what you have is essentially a promise to provide a specfication at some later date.  I don't think the IESG is going to be OK with that.

You have focusing on it being plausible in the sense of convincing people that this can be done
(i.e. that a definition along these lines can be produced) which it can, but I think the IESG is going to want you to show them that it has been done (i.e. that the definiton has been arrived at), and you are a ways away from being there.

The problem isn’t that we are pushing off a specification to a later date, but rather that the security is implementation specific. The client and MDS are forced to pick the same one.

Section 1.7.1 RFC 5661 is quite clear that:

   As with previous versions of NFS, the External Data Representation
   (XDR) and Remote Procedure Call (RPC) mechanisms used for the NFSv4.1
   protocol are those defined in [2] and [3].  To meet end-to-end
   security requirements, the RPCSEC_GSS framework [4] is used to extend
   the basic RPC security.  With the use of RPCSEC_GSS, various
   mechanisms can be provided to offer authentication, integrity, and
   privacy to the NFSv4 protocol.  Kerberos V5 is used as described in
   [5] to provide one security framework.  With the use of RPCSEC_GSS,
   other mechanisms may also be specified and used for NFSv4.1 security.

Note that nothing is stated as to how the TGT are constructed, so Ben’s approach:

meets the requirements of RFC 5661 and the loosely coupled model. The details
of how the mds gets the storage device’s keytab are up to the implementation
(just as how the storage device gets the keytab in the first place).

We could wait until the first implementation and then make that a standard, but I
don’t like that approach.

---

From: Olga Kornievskaia <[email-masked]>
Date: Wed, 2 Aug 2017 12:03:34 -0400

On Tue, Aug 1, 2017 at 3:59 PM, Thomas Haynes <[email-masked]>
wrote:

>
> On Aug 1, 2017, at 11:29 AM, David Noveck <[email-masked]> wrote:
>
> I think the treatment of security with tight coupling is sufficiently
> clear to implement from and has been for a while.  With regard to security
> for loose coupluing, what you have is essentially a promise to provide a
> The problem isn’t that we are pushing off a specification to a later date,
> but rather that the security is implementation specific. The client and MDS
> are forced to pick the same one.
>

I think you would agree that implementations frequently find issues with
this spec. But yes, all the previous specs when ahead prior to full
implementations. And that's why we have versions, right? It'll be fixed
later.

I'm thinking like an implementor and I have questions that cause me
problems. Let's for the moment assume that metadata server will be running
on the same machine as the normal KDC and have access to the principals
database. To construct these tickets you need a new service as it will not
be a part of the (existing) KDC. Let's assume it is practical.

Let's talk about of the fact that user doesn't know the password for the
synthetic identity.

Metadata server gets a flexile layout request. It, at some point,
constructs a service ticket  (session key, "synthetic identity", ...
)encrypted with data server's key. It needs to encrypt the "session key"
with the client's TGT key so that the client can then use the service
ticket against the data server. How doest the client gets this ticket?

In normal kerberos, the client would initiate request for a service ticket.
But here the client doesn't know the "synthetic identity" ahead of time.
And even if the intent is that once the ffds_owner is returned, then the
client would request a service ticket for the specified identity. Well in
that case, the client would need to provide the password for the synthetic
identity. But he doesn't know the password so that's not going to work. So
at this point we can't use normal kerberos to get a service ticket for the
synthetic identity.

Initially, in some earlier drafts of the spec the creds was "opaque_auth"
but it's not no longer there so I don't see a way of passing the service
ticket, encrypted session key. So I think if you want to pass the service
ticket back, then you need to add the "opaque_auth" back to the structure?
Ok now, we are just passing them back, how are we going to protect against
replay attacks? typically there is a challenge response. So perhaps require
that LAYOUTGET is always done with krb5i/p?

Also on the (linux) client side I would suspect you won't be able to use
standard krb5 libraries to deal with the processing the "service ticket".
You might have issues getting the gssd given that the identity inside the
ticket doesn't "match" the running identity of the user (I'm not 100%sure
here). *i understand that this comment is really implementation specific
and not specs job*

Having said all that, I still feel like we are proposing an "SPKM" security
here. It will not be practical and later on would be removed. I'd say
AUTH_SYS or GSSv3 should be used.

And if you still want to keep Kerberos then I guess I'm proposing (a) add
"opaque_auth cred" back to the ff_data_server_4 and (b) require that
LAYOUTGET much be done with krb5i/p. (c) as suggested by Ben, issue
"service tickets" and explicitly specify what is being passed back, need to
dig out the Kerberos spec and write out how the "synthetic identity"
information is fitted into those Kerberos structures. I feel that you need
to do this because the spec isn't use standard Kerberos so can't just say
see Kerberos spec.  (d) explicitly specify that metadata must have access
to the principal database for the non-synthetic client and the storage
servers.

I wonder is "opaque_auth" the right structure? I think the spec defines it
upto 400bytes. Is that sufficient to capture service ticket + the encrypted
stuff for the client. I dunno.

You mentioned that RFC5661 doesn't state anything as to how TGT are
constructed. That's only true because it states the Kerberos v5 and cites
the spec is used. This proposal does not follow Kerberos v5 spec -- the
proposal does not engage in KRB_AP/REP sequence which is what kerberos is
-- and thus i feel it must specify what goes into the service ticket.

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 4 Aug 2017 20:43:29 +0000

Hi,

I’m going to top post to keep the focus on the problem at hand.

In a nutshell, Olga is correct that we are not going to be able to secure loosely coupled flex files.
The removal of “opaque_auth” prevents this from occurring.

And while I have a bastardized approach using RPCSEC_GSSv3 structured privilege assertions,
like we did for SSC - see Section 4.9.1. in RFC 7862, I think a better approach is to
document the failure of securing loosely coupled flex files v1 and create a flex files v2
which addresses a minor change in the XDR to:

(a) Fix the stateid issue for loosely coupled NFSv4.x
(b) Fix this issue for “opaque_auth” - which includes the issues raised by Olga at the end of
her email.

As to how we go about that, I see two scenarios:

(1) We produce a single standards track document with a request for:

     +-----------------------+-------+----------+-----+----------------+
     | Layout Type Name      | Value | RFC      | How | Minor Versions |
     +-----------------------+-------+----------+-----+----------------+
     | LAYOUT4_FLEX_FILES    | 0x4   | RFCTBD10 | L   | 1              |
     | LAYOUT4_FLEX_FILES_v2 | 0x6   | RFCTBD10 | L   | 1              |
     +-----------------------+-------+----------+-----+----------------+

(2) We produce two standards track documents, the first for LAYOUT4_FLEX_FILES
and the second much smaller document which would have a normative
reference to the first and encompass LAYOUT4_FLEX_FILES_v2. It would
have the delta XDR and the new required text.

What I am approaching the WG for is consensus on which approach to take. For
me, the first is less administrative headache and the second preserves a much
cleaner separation of the minor versioning.

Oh, and in case it is not clear, I am very appreciative of Olga’s review in
getting me to see the flaw in this document.

Thanks,
Tom

On Aug 2, 2017, at 9:03 AM, Olga Kornievskaia <[email-masked]<mailto:[email-masked]>> wrote:

On Tue, Aug 1, 2017 at 3:59 PM, Thomas Haynes <[email-masked]<mailto:[email-masked]>> wrote:

I think you would agree that implementations frequently find issues with this spec. But yes, all the previous specs when ahead prior to full implementations. And that's why we have versions, right? It'll be fixed later.

I'm thinking like an implementor and I have questions that cause me problems. Let's for the moment assume that metadata server will be running on the same machine as the normal KDC and have access to the principals database. To construct these tickets you need a new service as it will not be a part of the (existing) KDC. Let's assume it is practical.

Let's talk about of the fact that user doesn't know the password for the synthetic identity.

Metadata server gets a flexile layout request. It, at some point, constructs a service ticket  (session key, "synthetic identity", ... )encrypted with data server's key. It needs to encrypt the "session key" with the client's TGT key so that the client can then use the service ticket against the data server. How doest the client gets this ticket?

In normal kerberos, the client would initiate request for a service ticket. But here the client doesn't know the "synthetic identity" ahead of time.  And even if the intent is that once the ffds_owner is returned, then the client would request a service ticket for the specified identity. Well in that case, the client would need to provide the password for the synthetic identity. But he doesn't know the password so that's not going to work. So at this point we can't use normal kerberos to get a service ticket for the synthetic identity.

Initially, in some earlier drafts of the spec the creds was "opaque_auth" but it's not no longer there so I don't see a way of passing the service ticket, encrypted session key. So I think if you want to pass the service ticket back, then you need to add the "opaque_auth" back to the structure? Ok now, we are just passing them back, how are we going to protect against replay attacks? typically there is a challenge response. So perhaps require that LAYOUTGET is always done with krb5i/p?

Also on the (linux) client side I would suspect you won't be able to use standard krb5 libraries to deal with the processing the "service ticket". You might have issues getting the gssd given that the identity inside the ticket doesn't "match" the running identity of the user (I'm not 100%sure here). *i understand that this comment is really implementation specific and not specs job*

Having said all that, I still feel like we are proposing an "SPKM" security here. It will not be practical and later on would be removed. I'd say AUTH_SYS or GSSv3 should be used.

And if you still want to keep Kerberos then I guess I'm proposing (a) add "opaque_auth cred" back to the ff_data_server_4 and (b) require that LAYOUTGET much be done with krb5i/p. (c) as suggested by Ben, issue "service tickets" and explicitly specify what is being passed back, need to dig out the Kerberos spec and write out how the "synthetic identity" information is fitted into those Kerberos structures. I feel that you need to do this because the spec isn't use standard Kerberos so can't just say see Kerberos spec.  (d) explicitly specify that metadata must have access to the principal database for the non-synthetic client and the storage servers.

I wonder is "opaque_auth" the right structure? I think the spec defines it upto 400bytes. Is that sufficient to capture service ticket + the encrypted stuff for the client. I dunno.

You mentioned that RFC5661 doesn't state anything as to how TGT are constructed. That's only true because it states the Kerberos v5 and cites the spec is used. This proposal does not follow Kerberos v5 spec -- the proposal does not engage in KRB_AP/REP sequence which is what kerberos is -- and thus i feel it must specify what goes into the service ticket.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 4 Aug 2017 21:41:29 +0000

Well, first I'll mention that fattr4_owner and fattr4_owner_group don't
seem to be defined anywhere (not in the RFCs such as 5661 I grep'd).
Presumably, they are strings, so you can define anything you want in them,
can't you?

Having said that. I think all you should do is say "if support of RPCSEC_GSS
is required, tight coupling needs to be used".
I think Olga suggested this in her first post on the topic?

If someone really wants to support RPCSEC_GSS/Kerberos for a loose
coupled pNFS server, let them figure out exactly how to do it and
write that up at that time. (I believe it might be possible, but doubt
anyone will actually implement it. Just like no one implements security
mechanisms other than Kerberos.)

rick
________________________________________
From: nfsv4 <[email-masked]> on behalf of Thomas Haynes <[email-masked]>
Sent: Friday, August 4, 2017 4:43:29 PM
To: Olga Kornievskaia
Cc: [email-masked]
Subject: Re: [nfsv4] New wording of Security Section for Flex Files

Hi,

Thanks,
Tom

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 4 Aug 2017 21:45:17 +0000

On Aug 4, 2017, at 2:41 PM, Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:

Full disclosure, the reason I want to define it is that Primary Data does
want to implement it, which means we need to have a standard between
at least the client and metadata server.

rick

---

From: Olga Kornievskaia <[email-masked]>
Date: Mon, 7 Aug 2017 12:06:30 -0400

Sorry might be a stupid question but why do you need to wait for another
version to add back "opaque_auth/or something" that carries the security
information?

On Fri, Aug 4, 2017 at 4:43 PM, Thomas Haynes <[email-masked]>
wrote:

> On Aug 2, 2017, at 9:03 AM, Olga Kornievskaia <[email-masked]> wrote:
>
>
>
>> The problem isn’t that we are pushing off a specification to a later
>> date, but rather that the security is implementation specific. The client
>> and MDS are forced to pick the same one.
>>
> In normal kerberos, the client would initiate request for a service
> ticket. But here the client doesn't know the "synthetic identity" ahead of
> time.  And even if the intent is that once the ffds_owner is returned, then
> the client would request a service ticket for the specified identity. Well
> in that case, the client would need to provide the password for the
> synthetic identity. But he doesn't know the password so that's not going to
> work. So at this point we can't use normal kerberos to get a service ticket
> for the synthetic identity.
>
> Having said all that, I still feel like we are proposing an "SPKM"
> security here. It will not be practical and later on would be removed. I'd
> say AUTH_SYS or GSSv3 should be used.
>

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 7 Aug 2017 16:22:46 +0000

> On Aug 7, 2017, at 9:06 AM, Olga Kornievskaia <[email-masked]> wrote:
> 
> 
> Sorry might be a stupid question but why do you need to wait for another version to add back "opaque_auth/or something" that carries the security information? 

Not a stupid question, I’m just so used to dealing with it that I don’t explain it all of the time:

We’ve had flex files in the Linux client for over 2 years and it understands the ff_data_server4 structure
to be a given byte format.  So an older client implementation would not be able to work with
a server which added the security field to the data structure.

---

From: Olga Kornievskaia <[email-masked]>
Date: Mon, 7 Aug 2017 12:35:09 -0400

> a server which added the security field to the data structure.

Thanks! Makes sense now.