Subject: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

From: Mike Eisler <[email-masked]>
Date: Tue, 09 Nov 2004 18:29:18 -0800

My comments are around the security area.

It isn't clear why new authentication systems are being invented
here, when we alread have one: ONC RPC authentication flavors, and
specifically, RPCSEC_GSS. The i-d needs to justify its direction.

> 4.1 Object Storage Security
> 
> 
>   The object storage protocol relies on a cryptographically secure
>   capability to control accesses at the object storage devices.
>   Capabilities are generated by the metadata server, returned to the
>   client, and passed to the object storage device, which verifies
>   that the capability allows the requested operation.
> 
> 
>   Each capability is specific to a particular object, an operation
>   on that object, a byte range w/in the object, and has an explicit
>   expiration time.  The capabilities are signed with a secret key
>   that is shared by the object storage devices (OSD) and the metadata
>   managers.  Typically each OSD has a set of master keys and working
>   keys, and the working keys are rotated periodically under the
>   control of the metadata manager.  Clients do not have device keys
>   so they are unable to forge capabilities.  Capabilities need to
>   be protected from snooping, which can be done by using facilities
>   such as Ipsec to create a secure VPN that contains the clients,
>   the file server, and the storage devices.

Requiring IPsec to protect the capabilities strikes me as
overkill.

Why doesn't the protocol assign each user a session key
known to metadata server and OSD, and then use the session
key to compute an HMAC on some or all of the messages?
GSS-API would be the ideal way to wrap an HMAC.

> 4.2 File Security
> 
> 
>   The file storage protocol has the same security mechanism between
>   the client and metadata server as between the client and data server.
>   This implies that the files that store the data need the same ACL as
>   the metadata file that represents the "control point" for the file.
>   This ensures that access control decisions are consistent between
>   the metadata server and the data server.
> 
> 
>   One alternative that was briefly discussed was the introduction
>   of special file handles that essentially have the properties of
>   capabilities so they can be generated by the metadata servers and
>   checked by the data servers.  (Peter Corbett described "one shot"
>   file handles.)  To be effective, these need all the properties of a
>   capability so the data server can efficiently and securely enforce
>   the access control decisions made by the metadata manager.

The problem with opaque special file handles is that presumably each
user of a given file gets his own file handle. This means NFS clients
are going cache multiple copies of the same data.

Assuming the capability route is going to be taken (a huge assumption,
IMHO), why not introduce a new NFSv4 pseudo op, (say "PUT_CAP") which injects
a capability into the COMPOUND procedure's operations stream?

> 
>   [We need to elaborate on this section. We should be able to
>   leverage the NFSv4 GSS context between the client and the NFSv4
>   "Storage Devices".]

If this is the direection, then at minimum the paragraph previous
to this should be struck. But to beat the not quite dead horse,
it is unclear why RPCSEC_GSS can't be used as is.

> 4.3 Block Security
> 
> 
>   The block model relies on SAN-based security, and trusts that
>   clients will only access the blocks they have been directed to use.
>   In these systems, there may not need to be any additional security
>   information returned with the map.  There are LUN masking/unmapping
>   and zone-based security schemes that can be manipulated to fence
>   clients from each other's data.  These are fairly heavy weight
>   operations that are not expected to be part of the normal execution
>   path for pNFS. But, a metadata server can always fall back to these
>   mechanisms if it needs to prevent a client from accessing storage
>   (i.e., "fence the client".)

Here I can accept that RPCSEC_GSS-like system authenticating a blocks protocol
isn't going to happen, mostly because blocks protocols have no
provision for real authentication. That said, at the recent NAS conference,
when the issue was raised, I thought I'd understood Brent Welch
to say that capabilities could be added to the blocks protocol.
If so, couldn't a capability, mutually signed by the metadata server and the NFS
server, could be used?

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Wed, 10 Nov 2004 12:39:27 -0500

> specifically, RPCSEC_GSS. The i-d needs to justify its direction.

i don't get what you mean. new authentication systems are not being invented. 
the intent is to use and existing storage protocol security features, 
including rpcsec_gss when present (not block layout, not OSD layout, only file 
layout).

> GSS-API would be the ideal way to wrap an HMAC.

ok - but using an HMAC is a change to the storage protocol, the use of ipsec 
is not.

pNFS might need an operation to communicate the OSD capabilities from the 
NFSv4 server to the pNFS client....
> >   the metadata server and the data server.

this is easy. a file that is created over pNFS with a file layout (stripped 
across NFSv4 'data servers') will naturally all have the same acl.

> >   the access control decisions made by the metadata manager.

i don't see a need for special file handles, or to discuss capabilities in the 
file layout case.

> If so, couldn't a capability, mutually signed by the metadata server and the NFS

we can certainly influence storage protocols so that they play better with pNFS security, but at the end of the day, we take what we get which in the current block world is nothing. adding a capabilty to the block protocol is outside the scope of pnfs. running the block protocol over ipsec is once again a consideration.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 10 Nov 2004 13:21:54 -0800

> i don't get what you mean. new authentication systems are not being invented. 

Capabilities are new to NFSv4, so new systems are being invented.

> including rpcsec_gss when present (not block layout, not OSD layout, only file 

I didn't see a statement in the i-d that file layout would use
RPCSEC_GSS (no mention of RPCSEC_GSS at all).

> ok - but using an HMAC is a change to the storage protocol, the use of ipsec 

IPsec authenticates coarse grained entities, ONC RPC authenticates
fine grained entities.

That the storage protocol apparently cannot authenticate fine grained entities
is not an excuse for accepting weaker security in NFSv4. We spent many years
in the NFSv4 working group establishing that fine grained authentication
was the way we would go, and now we are being presented with arguments that
because other protocols can't do fine grained, NFS security should be
watered down.

> i don't see a need for special file handles, or to discuss capabilities in the 

Well you co-authored this i-d, and it is mentioned. :-)

> we can certainly influence storage protocols so that they play better with pNFS 
 > security, but at the end of the day, we take what we get which in the current
> block world is nothing. adding a capabilty to the block protocol is outside the 
 > scope of pnfs. running the block protocol over ipsec is once again a consideration.

First, this is inconsistent with what Brent W stated at the NAS conference.
Second, IETF owns the iSCSI storage protocol, and it is certainly within IETF's
power to re-define (in a backwards compatible manner) iSCSI so that we get more
than nothing. If you were thinking Fibre Channel, yes, there's probably
nothing we can do there (but we once thought we couldn't do ONC RPC over RDMA
with RPCSEC_GSS and we were proven wrong), but by the same token, if pNFS is to be a standards
track protocol, its relationship with FC is also absolutely nothing. Implementors
of pNFS who want to support FC can do so, but such implementations have
nothing to do with the pNFS standard.

So I think the pNFS work needs to include in its charter the necessary extensions
to the blocks protocols IETF owns in order to maintain NFSv4's current authentication
model. Otherwise, we might as well specify AUTH_SYS as mandatory to implement, since
for all intents in purposes, that's the strength of authentication the current
pNFS i-d is offering.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 11 Nov 2004 12:14:17 -0600

> My comments are around the security area.

This was a heated topic of discussion at the meeting yesterday.

I made comments similar to yours.

I offer the authors this advice:

1) Work out the security properties of your proposals, discuss in the
I-Ds, list.

2) Propose requirements for security properties of pNFS, discuss.

3) I highly recommend that you include a REQUIRED to implement
mode/configuration which truly is comparable, in terms of security
properties, to NFSv4.

4) Work out the non-security related problems of pNFS.  There be dragons
in the layout delegation, as discussed yesterday.

5) Be sure to explain why lesser security is good for some environments
(e.g., HPC?) if the trade off is for higher performance and there are
mitigations.  You will get major pushback on this, so be sure to sell it
well.

Expect a lot of pushback if you don't do these things or if your
proposals fail to pass the security laugh test.  Your protocol's
security properties are NOT comparable to NFSv4's, not if the client is
made to be responsible for enforcing ACLs as you (the presenters)
admitted yesterday -- if your later proposals include such modes make
sure you make not such statements.

Cheers,

Nico
--

---

From: Brent Welch <[email-masked]>
Date: Thu, 11 Nov 2004 16:35:00 -0800

I have clouded the issue by introducing files, object, and blocks,
and by explaining the object security model incorrectly.
I will attempt a better explanation at the end of this email.

>>>Mike Eisler said:

First, the main point of confusion is the introduction of different
storage protocols in addition to the pNFS extensions that affect
the protocol between the client and the NFSv4 server.  The pNFS protocol
as currently described covers the client -> NFSv4 protocol.  The
client also has a client -> storage protocol that it will use to
read and write data.  With the goal of making things concrete, let's
assume that this protocol is also NFSv4.  HOWEVER, that is not the
only possibility.  For the purposes of the pNFS extension, this
possibility is acknowledged by the use of type fields in data layouts
that indicate what that storage protocol is.  Again, for a concrete
protocol spec, we plan to define layout types that are used with
NFSv4 as the storage protocol.  

But, we really do not want to preclude
other storage protocols.  The people discussing this protocol for
the past year come from companies that do NFS servers, scientists that
use non-kernel filesystems like PVFS, companies that do object storage,
and companies that do SAN filesystems.   By keeping these other
storage protocols in mind when we design the extension, and putting
in the appropriate type fields to all for them, we think we are adding
more value to the extension.  We are not requiring (i.e., REQUIRED) that
all possible storage protocols be implemented.  But, we would really
like to allow them.

About security.  I know you all have worked hard to get good security
into NFSv4, and I think that's great.  We have no intention of watering
down the security of NFSv4.  I think Dave Noveck said the right thing,
that the requirement is that the data servers honor the ACL stored
on the metadata server.  That can be done by pushing ACLs out to
the data servers (in a fault tolerant way) or by having the data servers
pull ACL information and cache it. Currently, the protocol used
between the metadata server and the storage servers is out of scope
of the pNFS extension.  (We're calling this the 
"Storage Management Protocol", although that is probably an overloaded term)

In contrast to what Nico said,
we've never stated that the client must interpret and enforce ACLs.
Instead, what we said is that some storage protocols (e.g., used with
SAN filesystems) rely on physical security, protection zones in the
storage network, and, yes, some trust of the client code.  However,
please keep in mind that if someone wants to use an insecure data
storage protocol in conjunction with pNFS, it is not REQUIRED that
anyone else use it, nor should their use of it in any way affect
the security of pNFS used, e.g., with NFSv4 as the storage protocol.

About the object security model.  First, yes, this is different.  It
comes from Howard Gobioff's thesis work.  It is designed to be something
that is cryptographically strong yet simple to enforce at the storage
device.  The pNFS extension is not mandating that a new security model
be used by NFSv4.  But, ideally, the pNFS extension would support the
use of this security model when OSD/iSCSI is used as the storage protocol.

The basic ideas are: the metadata server and the object storage devices
share a secret key. Key provisioning is part of the storage management
protocol and is out of scope of pNFS.  Next, when the metadata server
returns a capability to a client, it also returns a signing key to
the client.  When the client presents operations to the OSD (object
storage device) it signs those commands with the signing key.  Plus,
a timestamp nonce is included in the command to prevent replay attacks.
Where this affects pNFS is that the signing key needs to be returned
from the metadata server to the client through a secure channel. We
can use the GSS infrastructure for this, but I think it places a new
demand on the return path.

--
Brent Welch
Software Architect, Panasas Inc
Delivering the premier storage system for scalable Linux clusters

www.panasas.com
[email-masked]

---

From: Julian Satran <[email-masked]>
Date: Thu, 11 Nov 2004 20:02:05 -0500

> IPsec authenticates coarse grained entities, ONC RPC authenticates

The point made was that IPsec after authtication - provides integrity 
guarantees (and privacy if requested) for the communication between the 
authenticated parties. Many of the GSSxxx provide those as options as 
well. IPsec offers alse very good protection against man in the middle and 
replay attacks and has the widest hardware support.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 11 Nov 2004 22:39:49 -0600

On Thu, Nov 11, 2004 at 04:35:00PM -0800, Brent Welch wrote:
[...]
> "Storage Management Protocol", although that is probably an overloaded term)

If you rely on a pNFS-server-to-storage protocol to obtain NFSv4
security semantics then you will have to specify that protocol.
Yesterday you were quite clear that you thought that that piece ought to
remain unspecified and a field for proprietary innovation.  See below.

> we've never stated that the client must interpret and enforce ACLs.

I wrote: "[...] not if the client is made to be responsible for
enforcing ACLs as you (the presenters) admitted yesterday [...]"

I did not say "interpret."  Below you write "[...] and, yes, some trust
of the client code."  I think this is consistent with "enforce" --
though that may not have been the best choice of words; I think it
serves my point, which is that the security properties of your preferred
(?) approach to pNFS differ markedly from NFSv4's.

> the security of pNFS used, e.g., with NFSv4 as the storage protocol.

It seems feasible to get NFSv4 security semantics in pNFS by using a
storage protocol which does not deal in volume blocks but in file blocks
and/or by some out-of-band ACL-and-layout communication between the
metadata servers and the storage servers.  Yes.  But your slides, as I
recall, did not actually say this, did not mention the block-oriented-
storage-protocols-break-NFSv4-security-semantics-or-oob-protocols-are-
required matter and actually termed the security of pNFS "comparable" to
NFSv4's.

Now, that could well be a slip, a compression error if you wish
(slideshows make us do that), which we could ignore.  BUT, you must at
least specify a required, implementable and deployable configuration
which has security properties which are truly comparable to NFSv4's.

> demand on the return path.

This is clearly not feasible with currently available off-the-shelf
iSCSI implementations, yet it is the existence of the such, it seems to
me, that spurs you to propose the use of iSCSI as the client<->storage
protocol for pNFS.

Cheers,

Nico
--

---

From: Brent Welch <[email-masked]>
Date: Mon, 15 Nov 2004 19:12:12 -0800

Let me try some simple logic.

My kitten has green eyes.  Therefore all kittens have green eyes.

Block storage protocols are insecure, 
therefore all storage protocols are insecure.

I would like to table all discussion of block protocol security
as being out of scope.  I think it clouds the issue and gets in
the way of working out an acceptable pNFS extension.

I am trying to communicate three points:

1. pNFS can use NFSv4 as the storage protocol, and should be just
as secure as NFSv4.  I think REQUIREing that NFSv4 be implemented
by all clients as one of their possible storage protocols is OK,
and makes perfect sense.

2. pNFS should allow different storage protocols.  These are OPTIONAL
storage protocols.  This is important to me because my preferred
storage protocol is object storage (iSCSI/OSD).  And, there are others
for whom other storage protocols are important (legacy NFS servers,
PVFS implementations, block storage on a SAN).  Harking back to the
alice in wonderland logic above, it should be OK to allow other
storage protocols w/out compromising pNFS when used with NFSv4 
or some other secure storage protocol.  Please don't misinterpret this.
OF COURSE if a customer chooses to use an insecure storage protocol
they are taking a risk.  But, that doesn't mean all customers are
taking the same risk because they don't have to use or provide those
insecure storage protocols in their environment.

3. Object storage has a secure protocol, but it does place a
demand on the pNFS client to NFSv4 server communication.
This is worth debating. It requires that capabilities and signing
keys be returned to the client over a secure channel so they cannot
be snooped.

>>>Nicolas Williams said:
 > > "Storage Management Protocol", although that is probably an overloaded 
ter
     m)

www.panasas.com
[email-masked]

---

From: Bill Sommerfeld <[email-masked]>
Date: Mon, 15 Nov 2004 22:23:18 -0500

> demand on the return path.

This may just be a terminology point:

 0) if you're talking about a symmetric secret-key based MIC/MAC, using
the verb "sign" will often confuse folks into thinking that you mean
public key.

 1) if you're talking about a public key signature, the usual mechanism
for this sort of delegation involves the signer (the pNFS client in this
case) generating a signature keypair, and the authority (pNFS metadata
server) authorizing that key to the verifier (storage server).

 2) there are other ways to delegate authority to a client besides
passing keys around which may be more robust in the face of accidental
disclosures.. signed authorizations can be revoked;  shared keys might
not be so easy to change in the event that a client is compromised.

					- Bill

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 15 Nov 2004 22:57:50 -0600

> therefore all storage protocols are insecure.

NFSv4 is not a block storage protocol.  Neither will pNFS be one either.

You're comparing kittens to fully grown dogs.

Or is pNFS an improvement on block storage?  I don't see it that way.
Perhaps you do and can persuade us of this view?  I do see pNFS as an
improvement on _NFS_.

Seriously, figure out what security (and operational) properties you
want a pNFS to provide, figure out what you can get with each storage
protocol option and let's see where this goes.

Also, do we really have enough info right now to figure out that, yeah,
pNFS w/ iSCSI can be made to work w/o mods to deployed iSCSI boxes?  I
suspect the answer is "no."  Further, I suspect the answer is that mods
will be needed.  Finally, I'd much rather see a pNFS solution that has
security properties that truly are comparable to NFSv4's.

> the way of working out an acceptable pNFS extension.

We're not trying to improve block storage protocols, but we're trying to
come up with a parallelized NFS protocol -- a much, _much_ different
beast that block storage protocols.

So what's in scope is: what sort of protocol should pNFS clients use to
talk to the storage.

And BTW, I've yet to argue that iSCSI needs improvements.

> by all clients as one of their possible storage protocols is OK,

This is my position.  (Though some client<->storage extensions may be
needed for some things, for all I know.)

> insecure storage protocols in their environment.

Perhaps this will fly past the IESG, perhaps not.  I think we should
inquire with them at some point (and their answer may be that they can't
yet answer so you just have to try to deliver a document to them and
see).  Of course, first we need WG consensus on this.

> keys be returned to the client over a secure channel so they cannot

NFSv4 as the storage protocol is like an OSD that knows how to evaluate
ACLs.  Alternatives include a pNFS-server<->storage protocol that
enables a client to do X, and your capabilities token approach is a
third approach.

Cheers,

Nico
--

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 16 Nov 2004 15:43:52 -0500

>by all clients as one of their possible storage protocols is OK,

Making a requirement that NFSv4 be implemented is only part
of the picture. It's reasonable, but somewhat arbitrary. The
question to answer is:

"What level of security is guaranteed by pNFS storage access?"

It's unclear how a client and server, authenticated and
communicating over an NFSv4 RPCSEC_GSS channel to the
pNFS server, can be assured that their storage access receives
similar protections, even if NFSv4 is used for the storage access.

Or if they can't receive this assurance, how they detect what
assurance they do have, and/or request another.

The current draft needs to address this.

Tom.

---

From: Craig Everhart <[email-masked]>
Date: Wed, 17 Nov 2004 11:50:45 -0500

> (?) approach to pNFS differ markedly from NFSv4's.

NFSv4 trusts the client code to some extent, and I think the same extent
for some of these pNFS data models.  The NFSv4 trust takes the form of
trusting the correct manipulation of the user's authentication/
authorization context, so that, for instance, user A on the client
doesn't get access to a file just because user B on the client has
that access.  It's going to be hard to prevent the requirement for
that level of trust.

I agree that using block-based access to data devices has traditionally
required a far greater level of trust in clients, and I think that
such trust would likely be required in a deployment of pNFS that used
that sort of access.  Probably not for object- or file-based deployments,
though.  And there's room for some fancy block-based access that doesn't
use the vanilla kind of SCSI but requires less client trust.  That's what
I assume that you would intend with new SCSI/iSCSI work, either defining
new
iSCSI or SCSI attributes, or implementing appropriate stuff that's already
there.  But that's different from the trusted access to block storage that
has traditionally characterized SCSI, that requires a larger level of trust
in the client to which you grant, say, FC access to devices.  Which I think
would be great to be able to represent as a data choice for pNFS.

            Craig
-

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 18 Nov 2004 16:47:06 -0600

On Thu, Brent Welch wrote:
...
> "Storage Management Protocol", although that is probably an overloaded term)

I am concerned by the statement that this storage management protocol
is out of scope.  The reason for my concern is two-fold.  The first is
that for NFSv4, we did something similar in that we made the
server-to-server replication/migration protocol a work item for
"later".  We then put in support for the client->server support but
without, it seems, ensuring that all of the issues were will thought
out; there have been presentations to the effect that at a minimum,
additional interpretation clarifications need to be made to have the
repl/migr support work reasonably well.  If the server-to-server work
had been done or at least drafted concurrently with the NFSv4
protocol, the issues we seem to face today, could have been avoided.

Secondly, this discussion appears to have a lot of the "how is it
coordinated" text and it concerns me that if the backend behavior and
capabilities are not reasonably understood that the client->server
result will be unsatisfactory.

Spencer

---

From: Julian Satran <[email-masked]>
Date: Fri, 19 Nov 2004 16:32:34 +0200

> specifically, RPCSEC_GSS. The i-d needs to justify its direction.

The Object store capability protects the storage from unathorized access 
without having to mainatin
duplicate ACL at each storage device. This is substantially different than 
the service provided by RPCSEC_GSS.

> GSS-API would be the ideal way to wrap an HMAC.
Using IPsec is an expedient way to have a secure channel protect
the integrity (not necesaricly the privacy) of everything transmitted. 
However Object Storage may use
other techniques (signed commands and/or commands and data when IPsec is 
not deployed or deemed too expensive (BTW IPSec is the "least expensive" 
security to implement if you ignore key deplyment intricacies and has the 
widest hardware support (critical a high speed).

---

From: Mike Eisler <[email-masked]>
Date: Sat, 20 Nov 2004 05:13:07 -0800

> by all clients as one of their possible storage protocols is OK,

Simply putting the requirement on the client does nothing to
ensure inter operability. Change "all clients" to "all
clients, metadata servers, and data servers",
and now we can intemperate.

Otherwise, pNFS will balkanize. It might
anyway, but in that case, it is shame on the
implementors instead of shame on the WG.

> storage protocols.  This is important to me because my preferred

Then we need to explain the security model, and how it differs from
non-pNFS, and how it differs from pNFS w/ NFSv4 as the storage
protocol. That's what Security Considerations mean. Or the different
storage protocols are not even OPTIONAL; they are unmentioned. We cannot
have it both ways.

Once we understand the security ramifications, we can have a proper
debate.

(As an aside, I suggest that debate take place at Connectathon '05, when
most of us will be present, as opposed to the next IETF, when most of
us won't be [few will be willing to spend 3 back to back weeks at
cthon and IETF]).

> keys be returned to the client over a secure channel so they cannot

This object storage protocol, which is a very complex
protocol based on the size of the PDF, is presented
as immutable with respect to its security model,
when it in fact is a work in progress.
If a major consumer of it, pNFS, demands better, why can't the
standards body that owns it accommodate us?