Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 10 Nov 2004 15:16:42 -0500

> > > 4.2 File Security
> > > 
> > > 
> > >   The file storage protocol has the same security mechanism
between
> > >   the client and metadata server as between the client and data
server.
> > >   This implies that the files that store the data need the same
ACL as
> > >   the metadata file that represents the "control point" for the
file.
> > >   This ensures that access control decisions are consistent
between
> > >   the metadata server and the data server.
> 
> this is easy. a file that is created over pNFS with a file layout
(stripped 
> across NFSv4 'data servers') will naturally all have the same acl.

Well when a SETATTR changed the ACL, it would have to propagated to
all of the metadata servers (via the storage management protocol) and,
if the server went down, you would need to deal with the issue of the 
ACL's being different on different stripes and how to repair that.

What is easy in prototype mode may not so easy in shipping product
mode, which is something we all have to deal with.

The pragraph quoted above suggests that the each of stripe files have
to have the same ACL and that can pose difficulties and I think that
they can avoided.  As written, the paragraph (seems to) suggest that 
the clients could do individual GETATTR(ACL)'s and it is a requirement
that the clients gets back an ACL that matches the one on the metadata
server.

I think the only requirement should be that the data servers respond to
IO requests in way which is consistent with the ACL set on the server.
If you want to not store the ACL on the data servers and have them get
the information on which principals have what access to which files 
from the metadata server, and cache that information and flush it when 
the metadata server tells them it changes, then that should be allowed. 

-----Original Message-----
From: William A.(Andy) Adamson [mailto:[email-masked]] 
Sent: Wednesday, November 10, 2004 12:39 PM
To: Mike Eisler
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt 

> My comments are around the security area.
> 
> It isn't clear why new authentication systems are being invented
> here, when we alread have one: ONC RPC authentication flavors, and
> specifically, RPCSEC_GSS. The i-d needs to justify its direction.

i don't get what you mean. new authentication systems are not being
invented. 
the intent is to use and existing storage protocol security features, 
including rpcsec_gss when present (not block layout, not OSD layout,
only file 
layout).

> 
> > 4.1 Object Storage Security
> > 
> > 
> >   The object storage protocol relies on a cryptographically secure
> >   capability to control accesses at the object storage devices.
> >   Capabilities are generated by the metadata server, returned to the
> >   client, and passed to the object storage device, which verifies
> >   that the capability allows the requested operation.
> > 
> > 
> >   Each capability is specific to a particular object, an operation
> >   on that object, a byte range w/in the object, and has an explicit
> >   expiration time.  The capabilities are signed with a secret key
> >   that is shared by the object storage devices (OSD) and the
metadata
> >   managers.  Typically each OSD has a set of master keys and working
> >   keys, and the working keys are rotated periodically under the
> >   control of the metadata manager.  Clients do not have device keys
> >   so they are unable to forge capabilities.  Capabilities need to
> >   be protected from snooping, which can be done by using facilities
> >   such as Ipsec to create a secure VPN that contains the clients,
> >   the file server, and the storage devices.
> 
> Requiring IPsec to protect the capabilities strikes me as
> overkill.
> 
> Why doesn't the protocol assign each user a session key
> known to metadata server and OSD, and then use the session
> key to compute an HMAC on some or all of the messages?
> GSS-API would be the ideal way to wrap an HMAC.

ok - but using an HMAC is a change to the storage protocol, the use of
ipsec 
is not.

pNFS might need an operation to communicate the OSD capabilities from
the 
NFSv4 server to the pNFS client....
> 
> > 4.2 File Security
> > 
> > 
> >   The file storage protocol has the same security mechanism between
> >   the client and metadata server as between the client and data
server.
> >   This implies that the files that store the data need the same ACL
as
> >   the metadata file that represents the "control point" for the
file.
> >   This ensures that access control decisions are consistent between
> >   the metadata server and the data server.

this is easy. a file that is created over pNFS with a file layout
(stripped 
across NFSv4 'data servers') will naturally all have the same acl.

> > 
> > 
> >   One alternative that was briefly discussed was the introduction
> >   of special file handles that essentially have the properties of
> >   capabilities so they can be generated by the metadata servers and
> >   checked by the data servers.  (Peter Corbett described "one shot"
> >   file handles.)  To be effective, these need all the properties of
a
> >   capability so the data server can efficiently and securely enforce
> >   the access control decisions made by the metadata manager.

i don't see a need for special file handles, or to discuss capabilities
in the 
file layout case.

> The problem with opaque special file handles is that presumably each
> user of a given file gets his own file handle. This means NFS clients
> are going cache multiple copies of the same data.
> 
> Assuming the capability route is going to be taken (a huge assumption,
> IMHO), why not introduce a new NFSv4 pseudo op, (say "PUT_CAP") which
injects
> a capability into the COMPOUND procedure's operations stream?
> 
> > 
> >   [We need to elaborate on this section. We should be able to
> >   leverage the NFSv4 GSS context between the client and the NFSv4
> >   "Storage Devices".]
> 
> If this is the direection, then at minimum the paragraph previous
> to this should be struck. But to beat the not quite dead horse,
> it is unclear why RPCSEC_GSS can't be used as is.
> 
> > 4.3 Block Security
> > 
> > 
> >   The block model relies on SAN-based security, and trusts that
> >   clients will only access the blocks they have been directed to
use.
> >   In these systems, there may not need to be any additional security
> >   information returned with the map.  There are LUN
masking/unmapping
> >   and zone-based security schemes that can be manipulated to fence
> >   clients from each other's data.  These are fairly heavy weight
> >   operations that are not expected to be part of the normal
execution
> >   path for pNFS. But, a metadata server can always fall back to
these
> >   mechanisms if it needs to prevent a client from accessing storage
> >   (i.e., "fence the client".)
> 
> Here I can accept that RPCSEC_GSS-like system authenticating a blocks
protocol
> isn't going to happen, mostly because blocks protocols have no
> provision for real authentication. That said, at the recent NAS
conference,
> when the issue was raised, I thought I'd understood Brent Welch
> to say that capabilities could be added to the blocks protocol.
> If so, couldn't a capability, mutually signed by the metadata server
and the NFS
> server, could be used?

we can certainly influence storage protocols so that they play better
with pNFS security, but at the end of the day, we take what we get which
in the current block world is nothing. adding a capabilty to the block
protocol is outside the scope of pnfs. running the block protocol over
ipsec is once again a consideration.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 11 Nov 2004 10:13:55 -0500

> > > including rpcsec_gss when present (not block layout, not 
> > OSD layout, only file 
> > > layout).
> > 
> > I didn't see a statement in the i-d that file layout would use
> > RPCSEC_GSS (no mention of RPCSEC_GSS at all).
> 
> I understand that requirement for NFSv4 based data servers but
> I can easily imagine someone wanting to cut the cost of the
> data servers and run NFSv3 on them rather than NFSv4.

I can imagine lots of things that someone might want, but I think
that if we try to approach pNFS trying to accommodate all such
possibilities we are not going to make much progress in going
forward providing facilities that people do in fact want, and
want badly.

We already have three proposed variants of pNFS and, because of 
hardware considerations, we may wind up having sub-variants (e.g. 
iSCSI and FCP) but a proliferation of sub-variants such as we 
would have if we tried to provide a standard definition of pNFS 
over v3 or even pNFS over v2 (with a layout each subfile could 
be kept under 4GB) is not a good thing.  More variants means 
more work in implementation and testing and a lot less effective 
interoperability as seen by the user.

We have talked about pNFS having provisions for vendor-specific
and experimental storage protocols and if you want to use v3 in
that framework, go ahead.  But as far as something that the working
group will define, I would argue very strongly that we limit ourself
to one storage protocol of the file variety at a time and only start
on a second one when pnfs with the first becomes a Proposed Standard.

Now, as Mike has pointed out, RPCSEC_GSS is available in v3, and even
if we did try to standardize v3 as pNFS storage protocol (which I
don't think we should do), RPCSEC_GSS would be available as a security 
mechanism.  So I think Mike's point that RPCSEC_GSS should be mentioned 
with regard to security for the files variant is a valid one.

-----Original Message-----
From: Halevy, Benny [mailto:[email-masked]]
Sent: Thursday, November 11, 2004 4:51 AM
To: 'Mike Eisler'
Cc: William A.(Andy) Adamson; [email-masked]
Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

Mike Eisler wrote:
> 
> William A.(Andy) Adamson wrote:
> 
> > i don't get what you mean. new authentication systems are 
> not being invented. 
> 
> Capabilities are new to NFSv4, so new systems are being invented.

Capabilities are mentioned in the context of object storage.
For pNFS over files it is mentioned as "One alternative that was briefly
discussed was the introduction of special file handles that essentially
have the properties of capabilities"

> > including rpcsec_gss when present (not block layout, not 
> OSD layout, only file 
> > layout).
> 
> I didn't see a statement in the i-d that file layout would use
> RPCSEC_GSS (no mention of RPCSEC_GSS at all).

I understand that requirement for NFSv4 based data servers but
I can easily imagine someone wanting to cut the cost of the
data servers and run NFSv3 on them rather than NFSv4.

> > ok - but using an HMAC is a change to the storage protocol, 
> the use of ipsec 
> > is not.
> 
> IPsec authenticates coarse grained entities, ONC RPC authenticates
> fine grained entities.
> 
> That the storage protocol apparently cannot authenticate fine 
> grained entities
> is not an excuse for accepting weaker security in NFSv4. We 
> spent many years
> in the NFSv4 working group establishing that fine grained 
> authentication
> was the way we would go, and now we are being presented with 
> arguments that
> because other protocols can't do fine grained, NFS security should be
> watered down.
> 
> >> >   file handles.)  To be effective, these need all the 
> properties of a
> > i don't see a need for special file handles, or to discuss 
> capabilities in the 
> > file layout case.
> 
> Well you co-authored this i-d, and it is mentioned. :-)
> 
> >> Here I can accept that RPCSEC_GSS-like system 
> authenticating a blocks protocol
> >> provision for real authentication. That said, at the 
> recent NAS conference,
> >> If so, couldn't a capability, mutually signed by the 
> metadata server and the NFS
> >> server, could be used?
> >> 
> > 
> > we can certainly influence storage protocols so that they 
> play better with pNFS 
>  > security, but at the end of the day, we take what we get 
> which in the current
> > block world is nothing. adding a capabilty to the block 
> protocol is outside the 
>  > scope of pnfs. running the block protocol over ipsec is 
> once again a consideration.
> 
> First, this is inconsistent with what Brent W stated at the 
> NAS conference.
> Second, IETF owns the iSCSI storage protocol, and it is 
> certainly within IETF's
> power to re-define (in a backwards compatible manner) iSCSI 
> so that we get more
> than nothing. If you were thinking Fibre Channel, yes, 
> there's probably
> nothing we can do there (but we once thought we couldn't do 
> ONC RPC over RDMA
> with RPCSEC_GSS and we were proven wrong), but by the same 
> token, if pNFS is to be a standards
> track protocol, its relationship with FC is also absolutely 
> nothing. Implementors
> of pNFS who want to support FC can do so, but such 
> implementations have
> nothing to do with the pNFS standard.

I'm all for improving the security mechanisms possible with
pNFS and have the infrastructure in the protocol to allow
such extensions in the future, if possible without revising
the protocol, but rather with versioning the storage protocol
variants.  I'd be careful not to raise the bar too high initially
though.  It is a big win if we can have pNFS working with standard
protocols available today.

> 
> So I think the pNFS work needs to include in its charter the 
> necessary extensions
> to the blocks protocols IETF owns in order to maintain 
> NFSv4's current authentication
> model. 

Agreed.

> Otherwise, we might as well specify AUTH_SYS as 
> mandatory to implement, since
> for all intents in purposes, that's the strength of 
> authentication the current
> pNFS i-d is offering.
> 
>

---

From: "Halevy, Benny" <[email-masked]>
Date: Thu, 11 Nov 2004 10:24:30 -0500

Mike Eisler wrote:
> Halevy, Benny wrote:
> Which WG is standardizing object storage? Where can one examine the
> protocol specifications?

OSD is a T10 SCSI command set, see
http://www.t10.org/ftp/t10/drafts/osd2/osd2r00.pdf.

> RPCSEC_GSS runs on NFSv[234]. I've kind of given up repeating this
> fact outside the WG, since Kerberos V5 is never associated with
> NFSv3 among NFS users, but I still have hope the working group
> members will understand this someday. :-)
> 
> Anyway, the i-d starts off as:
> 
>   "The pNFS extension to NFSv4 takes the form of new operations that
>    return data location information called a "layout"."
> 
> The i-d specifies a bunch of new operations. How are these
> operations going to be added to NFSv3? New procedures? What
> if someone else decides they want to add new procedures with
> the same numbers?
> 

Hmm, what I had in mind was a (p)NFSv4 front end ("metadata server")
and NFSv3 based back end servers ("data servers") with no new operations
to NFSv3.

---

From: "Halevy, Benny" <[email-masked]>
Date: Thu, 11 Nov 2004 10:32:45 -0500

Noveck, Dave wrote:
> that framework, go ahead.  

That was exactly my point - not to rule out such extensions within
the framework.

> But as far as something that the working

good point.

> with regard to security for the files variant is a valid one.

agreed.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 15 Nov 2004 09:33:29 -0500

But such a server is going to have to be taught some storage management protocol
and may need to be upgraded to enforce ACL's, implement Kerberos, etc.  It may be
that teaching a v3 server these additional things it does not now know, might wind up 
being simpler for some significant group of people, than providing a general upgrade for 
the data repository software including v4 support, but that remains to be seen.  Color
me unconvinced.
 
Clearly, within the pnfs framework, you could support v3 as the storage access protocol,
and I'm not proposing that we do anything to foreclose it.  But pnfs already has a problem
as far as fragmentation goes in that it is attempting to support files, objects, and blocks
as well.  Trying to do more than one storage access protocol within the files area at one
time is a recipe for delay.  The implementation for v2/v3 might turn out to be straightforward,
but the specification almost certainly will not.  For example, when v3 is used as pnfs 
storage access protocol, is Kerberos mandatory to implement?  How about SPKM3
and LIPKEY?   Achieving consensus will take time and effort. 
 
Let's focus for now on v4 itself as a storage access protocol, and if that work gets done, 
we can consider other file-flavored storage access protocols.  Anyone for CIFS? :-)

-----Original Message-----
From: Julian Satran [mailto:[email-masked]]
Sent: Monday, November 15, 2004 8:41 AM
To: Noveck, Dave
Cc: Mike Eisler; [email-masked]
Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

[email-masked] wrote on 12/11/2004 17:20:51:

> > I think that adding data access features is better than 
> > replicating ACLs 
>   
> Not sure what you are saying here. 
>   
> > but I doubt that excluding V2 & V3 from data access makes too much 
> > sense giving the long time required for a new protocol to become 
> > prevalent and the widespread use of V2/V3 
>   
> I don't understand your logic here.  The set of pNFS clients is already 
> limited to systems that can speak v4.x (for whatever x turns out to be). 
> So the widespread use of V2/V3 by clients that cannot speak v4.x does 
> not really come into it.  They will not be able to speak pNFS, whether 
> we do the work to support V2 and V3 as a storage protocol or not.  On 
> the other hand, since all clients which support pNFS must have a V4.x 
> client (or else they wouldn't be pNFS clients), supporting v4.x as a 
> storage protocol should provide 100% coverage of that subset (unless 
> the implementer has some ideological objection to the files variant). 
>   
> It is true that pNFS will take time to become prevalent but that comes 
> with the territory.  If you want new features that aren't provided by 
> an existing protocol, you have to wait for that protocol to happen. 
>   
> No matter what we do about storage protocols, V2 and V3 clients would be 
> supported by pNFS servers in proxy mode, the same way they would support 
> V4.0 clients or V4.x clients that don't have pNFS support. 
> 

I was thinking more about supporting V2/V3 servers as data repositories 
and accessing them from a pNFS client should be straightforward 

> From: Julian Satran [mailto:[email-masked]]
> Sent: Thursday, November 11, 2004 8:02 PM
> To: Noveck, Dave
> Cc: Mike Eisler; [email-masked]; [email-masked]

> 
> 
> [email-masked] wrote on 10/11/2004 10:58:53:
> 
> > Apart from the merits of the capability approach, Mike's suggestion
> > is a specific instance of a general point, which we should take
> > note of.  If nfsv4 is going to be used as a storage protocol within 
> > pNFS we have the option of adding features specifically designed
> > to support that use.  If pNFS is going appear in a specific v4.x,
> > I would argue that it is quite reasonable only to support v4.x 
> > (and not earlier v4 versions) as the pNFS storage access protocol.
> > 
> 
> I think that adding data access features is better than replicating 
> ACLs but I doubt that excluding V2 & V3 from data access makes too 
> much sense giving the long time required for a new protocol to 
> become prevalent and the widespread use of V2/V3 
> 
> > From: Mike Eisler [mailto:[email-masked]] 
> > Sent: Tuesday, November 09, 2004 9:29 PM
> > To: [email-masked]
> > Subject: [nfsv4] comments on draft-welch-pnfs-ops-00.txt
> > 
> > 
> > >   that is shared by the object storage devices (OSD) and the metadata
> > >   This implies that the files that store the data need the same ACL as
> > >   the metadata file that represents the "control point" for the file.
> > >   file handles.)  To be effective, these need all the properties of a
> > >   clients will only access the blocks they have been directed to use.
> > >   information returned with the map.  There are LUN masking/unmapping
> > >   operations that are not expected to be part of the normal execution
> > >   path for pNFS. But, a metadata server can always fall back to these
> > If so, couldn't a capability, mutually signed by the metadata server and
> > the NFS
> > server, could be used?
> > 
> > 
> > 
> > 
> > 
> > 
> >

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 16 Nov 2004 09:15:56 -0500

> But, we really do not want to preclude
> other storage protocols.  The people discussing this protocol for
> the past year come from companies that do NFS servers, scientists that
> use non-kernel filesystems like PVFS, companies that do object
storage,
> and companies that do SAN filesystems.   By keeping these other
> storage protocols in mind when we design the extension, and putting
> in the appropriate type fields to all for them, we think we are adding
> more value to the extension.  We are not requiring (i.e., REQUIRED)
that
> all possible storage protocols be implemented.  But, we would really
> like to allow them.

I may be the victim of a misspent youth studying mathematical logic,
but it seems to me that the cardinality of the set of "all possible
storage protocols" is aleph-zero.  I think we have to draw the line
somewhere short of that and I'm going to have real trouble with 
having more than 4 billion storage protocols :-)  The storage 
protocol indicator should be a (no more than) 32-bit integer, and
not a list of integers. :-)

Seriously, I'm OK with trying not to preclude additional storage 
protocols, as long as we don't have to try too hard.  If we start
building up any significant list of items needed to accommodate
future storage protocols, then my inclination would be to leave 
future storage protocols to the future and move ahead with the
small set we want most.  I agree that allowing additional storage
protocols does add value, but for me, that additional value is 
far less than that of getting pnfs done and approved with a small 
(non-empty) set of storage protocols.  A singleton set of storage
protocols at time T is a lot better than a large set at T+1 year. 

-----Original Message-----
From: Brent Welch [mailto:[email-masked]] 
Sent: Thursday, November 11, 2004 7:35 PM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt 

I have clouded the issue by introducing files, object, and blocks,
and by explaining the object security model incorrectly.
I will attempt a better explanation at the end of this email.

>>>Mike Eisler said:

First, the main point of confusion is the introduction of different
storage protocols in addition to the pNFS extensions that affect
the protocol between the client and the NFSv4 server.  The pNFS protocol
as currently described covers the client -> NFSv4 protocol.  The
client also has a client -> storage protocol that it will use to
read and write data.  With the goal of making things concrete, let's
assume that this protocol is also NFSv4.  HOWEVER, that is not the
only possibility.  For the purposes of the pNFS extension, this
possibility is acknowledged by the use of type fields in data layouts
that indicate what that storage protocol is.  Again, for a concrete
protocol spec, we plan to define layout types that are used with
NFSv4 as the storage protocol.  

But, we really do not want to preclude
other storage protocols.  The people discussing this protocol for
the past year come from companies that do NFS servers, scientists that
use non-kernel filesystems like PVFS, companies that do object storage,
and companies that do SAN filesystems.   By keeping these other
storage protocols in mind when we design the extension, and putting
in the appropriate type fields to all for them, we think we are adding
more value to the extension.  We are not requiring (i.e., REQUIRED) that
all possible storage protocols be implemented.  But, we would really
like to allow them.

About security.  I know you all have worked hard to get good security
into NFSv4, and I think that's great.  We have no intention of watering
down the security of NFSv4.  I think Dave Noveck said the right thing,
that the requirement is that the data servers honor the ACL stored
on the metadata server.  That can be done by pushing ACLs out to
the data servers (in a fault tolerant way) or by having the data servers
pull ACL information and cache it. Currently, the protocol used
between the metadata server and the storage servers is out of scope
of the pNFS extension.  (We're calling this the 
"Storage Management Protocol", although that is probably an overloaded
term)

In contrast to what Nico said,
we've never stated that the client must interpret and enforce ACLs.
Instead, what we said is that some storage protocols (e.g., used with
SAN filesystems) rely on physical security, protection zones in the
storage network, and, yes, some trust of the client code.  However,
please keep in mind that if someone wants to use an insecure data
storage protocol in conjunction with pNFS, it is not REQUIRED that
anyone else use it, nor should their use of it in any way affect
the security of pNFS used, e.g., with NFSv4 as the storage protocol.

About the object security model.  First, yes, this is different.  It
comes from Howard Gobioff's thesis work.  It is designed to be something
that is cryptographically strong yet simple to enforce at the storage
device.  The pNFS extension is not mandating that a new security model
be used by NFSv4.  But, ideally, the pNFS extension would support the
use of this security model when OSD/iSCSI is used as the storage
protocol.

The basic ideas are: the metadata server and the object storage devices
share a secret key. Key provisioning is part of the storage management
protocol and is out of scope of pNFS.  Next, when the metadata server
returns a capability to a client, it also returns a signing key to
the client.  When the client presents operations to the OSD (object
storage device) it signs those commands with the signing key.  Plus,
a timestamp nonce is included in the command to prevent replay attacks.
Where this affects pNFS is that the signing key needs to be returned
from the metadata server to the client through a secure channel. We
can use the GSS infrastructure for this, but I think it places a new
demand on the return path.

--
Brent Welch
Software Architect, Panasas Inc
Delivering the premier storage system for scalable Linux clusters

www.panasas.com
[email-masked]

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 16 Nov 2004 16:59:34 -0500

> On Tue, Nov 16, 2004 at 01:14:46PM -0500, Noveck, Dave wrote:
> > So the question is, how do you create a meaningful requirement?  
> > OK, it's mandatory to provide a NFSv4-style layout.  Nothing could
> > be simpler.  The layout for file handle FH says that the data is
> > at FH' and IP address X (one of the servers IP addresses) and your
> > done.  Is that OK?  The performance of accessing the data using 
> > that layout will be no better than accessing it over standard 
> > NFSv4, but do we want to get in the business of making performance
> > requirements?
>
> Huh?  Why couldn't the meta-data server say
> 
> [{fs_location1, FH', byte-ranges}, {fs_location2, FH', byte-ranges}, ..,
>  {fs_locationN, FH', byte-ranges}]

It could if the data repositories were NFS boxes.  If they were SAN
device (where the design center was SAN as a storage protocol), then
they couldn't do this meaningfully and so a single location and a 
single FH' would bve the easiest layout to give out.  And it is a 
valid one.  If you want to say that that's cheating, I guess I 
would agree but I can't some up with a deifnition of "cheating"
that one could put in spec that one couldn't get around.

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]]
Sent: Tuesday, November 16, 2004 1:31 PM
To: Noveck, Dave
Cc: Brent Welch; Mike Eisler; [email-masked]
Subject: Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

On Tue, Nov 16, 2004 at 01:14:46PM -0500, Noveck, Dave wrote:
> So the question is, how do you create a meaningful requirement?  
> OK, it's mandatory to provide a NFSv4-style layout.  Nothing could
> be simpler.  The layout for file handle FH says that the data is
> at FH' and IP address X (one of the servers IP addresses) and your
> done.  Is that OK?  The performance of accessing the data using 
> that layout will be no better than accessing it over standard 
> NFSv4, but do we want to get in the business of making performance
> requirements?

Huh?  Why couldn't the meta-data server say

[{fs_location1, FH', byte-ranges}, {fs_location2, FH', byte-ranges}, ..,
 {fs_locationN, FH', byte-ranges}]

And if OSD-like extensions to NFSv4 are needed that's fine with me.

>                You can argue that this is an evasion of the
> requirment but it just isn't possible to draw the line between
> evasion and avoidance in a way that sticks.  So a client would
> learn to ignore this sort of layout, and use the ones that that
> server makes faster.  In which case, why should we ask the server
> to pretend?  If he is set up so that pnfs-blocks is fast, then 
> why should we ask him to tell the client that pnfs-files is 
> available, when it isn't, for all practical purposes.

See above.  I do think that NFSv4 may need extending to make it a
high-performance proposition for this.

> > > The reason that we can't make this out of scope is that the proposal
> > > which is under discussion is intended to allow block protocols.  If
> > > block protocols were not possible, because of valid security issues,
> > > then the base protocol would look different.  So, even though we 
> >                                                 ^^^^^^^^^^^^^^^^^^
> > > can discuss such security issues, the bar for those raising these
> >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > security objections is very high.  I think would have to show that no
> >   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > possible block protocol could be acceptable (or at least worthwhile), 
> > > which is a very hard thing to show (since it isn't true or at least 
> > > I don't think it is).
> 
> > Hooey.  The NFSv4 WG charter says nothing about pNFS, so I think all
> > aspects of any pNFS proposals are up for discussion until consensus is
> > reached.
> 
> Sure, but your right to discuss things has to deal with my (and others')
> right to decide what arguments to listen to.  What I guess I was saying
> in the above sentence is that for me, the appropriate time to discuss
> the security flaws of a particular proposal to use a blocks protocol
> is when such a proposal is actually made, i.e. there is a document that 
> really defines what is proposed.  Until then, we are dealing with 
> stuff that is speculative, unless you can show me that it is impossible
> that any proposal in this vein will be acceptable.   

The authors termed the security of pNFS "comparable" to NFSv4's -- in
their slides at last week's meeting.  No, I have not read their
document, but this is enough for me to get concerned.  Here I start to
repeat myself; I'll not and instead I offer to review the new version of
the pNFS I-D(s).

> > Anyways, the burden is on the authors.  Let them show us the various
> > combinations they have in mind and let them tell us what their security
> > and operational properties are.
> 
> I agree, once they want to make specific proposals on additional storage
> protocols.  For me, saying that objects are interesting and that the
> general pnfs framework should accommodate blocks protocols, does not 
> count as making a specific proposal.

I will now shut up and wait for the -01.  We're close to going in
circles.

The authors now have several copies of my comments on what I'd like to
see from them, as well as lots of comments from others, including
Mike's, who started the whole pNFS security discussion.

Over to the authors.

Cheers,

Nico
--

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Wed, 17 Nov 2004 12:53:55 -0500

[email-masked] said:
> I for one disagree very strongly.  Although it may sound like it,
> "pNFS" is not a separate protocol.  It is a set of extensions to  v4
> and a pNFS server is an NFSv4 server with some optional features
> enabled (just as soon as we define those).  Not acting like a normal
> v4 server in response to a read (for example) probably violates the
> minor versioning rules (can't say for sure without all the details)
> but it certainly violates the spirit.  We shouldn't go there.

so, in order to act like a normal v4 server,  should pNFS require the storage 
protocol to check the stateid presented in the NFSv4 READ and WRITE?

-->Andy

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 17 Nov 2004 13:39:42 -0500

I was going to point out that a model in which you took the fh directly
from the 
meta-data was closer to that we would have with block and object storage
protocols,
than one that supposed that directories were present and opens processed
on the
data repository.  But I didn't because I wasn't sure whether that would
be judged a 
plus or a minus :-)
 
As to the attribute stuff, I do think that the attributes are the
property/responsibility
of the meta-data server (although single-stripe layouts, aka file
referrals, may suggest
we want to allow other options), I think the current LAYOUT COMMIT is
really clunky.
Although it is necessary for blocks, it seems better to me for the
layout to carry an
indication of whether LAYOUT COMMIT needs to be done by the client.
Where it 
isn't, it would be the responsibility of the data repository to
propagate eof and change/
access time updates to the meta-data server.

	-----Original Message-----
	From: Craig Everhart [mailto:[email-masked]] 
	Sent: Wednesday, November 17, 2004 10:55 AM
	To: [email-masked]
	Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

	Dave Noveck <[email-masked]> wrote on 11/17/2004
09:46:28 AM:
	
	> Slightly longer answer: I think we have to understand better
how locking,
	> including byte-range locking (possibly mandatory) will fit
into the model.
	> 
	> Some possiblities are that the meta-data server assigns it as
part of the
	> layout, that it is obtained via an fh-based variant of open,
and that it is
	> a fixed special stateid.
	> 
	> The basic reason for a stateid is so that IO operations can be
tagged with
	> what is essentially an owner so that the server can determine
if they 
	> conflict with a lock held by another or are being done by the
same open/lock
	> owner and are therefore exempt from such a check.
	> 
	> Now I assume that share reservations are checked at the
meta-data server and
	> even if you do an OPEN nobody is going to be doing them on a
region with 
	> other than DENY=NONE.  I don't think we've talked about where
byte-range
	> locking is done, but I wasn't expecting that the data
repository would 
	> have these and even if it did there would be some that crossed
the region
	> boundaries.
	> 
	> So to me the first question is whether we need a stateid.  If
we do, I'm
	> not clear that we want to force the client into having two
more round trips
	> to each of the data servers to get them, and even if that is
the case, I
	> don't like the implementation complexity caused by forcing the
implementer
	> to duplicate directory structure on multiple machines and keep
them in sync.
	> Maybe we will be forced to do that but I would want to explore
lot of 
	> alternatives before I went there.
	
	My model for this is based on block storage, in which everything
other than
	read and write is handled by the metadata server.  Thus, you
open the file
	on the metadata server, you get byte-range locks from the
metadata server.
	If you need mandatory locks, you get them from the metadata
server.
	The referenced third-party data source is used *only* for read
and write.
	Share reservation collisions are handled by interactions with
the metadata
	server.
	
	By the way, the pNFS model prompted by FC block storage also
suggests that
	file attributes like EOF also reside with the metadata, so that
GETATTR
	and SETATTR that deals in file length are strictly
metadata-server operations,
	not data-server operations, as well as byte-range locks and
share/deny
	reservations.  Does this jive with the pNFS models prompted by
the other
	two architectures for third-party storage (objects and files)?
Seems like
	it would be OK--after all, the other third-party storage models
are usually
	composed of multiple sub-things (multiple objects, multiple
files, striped or
	concatenated or whatever), so that the whole-file attributes
(like whole-file
	EOF) and share/deny reservations need to be dealt with by the
central metadata
	server for that whole file.
	
	Yes, yes, we could also get a lot more clever and design more
kinds of
	third-party storage, including some based on iSCSI so it fits
more obviously 
	under the IETF umbrella, with just the properties one
desires--perhaps
	object-inspired security, byte range locking (advisory or
mandatory), whatever.
	It's a potentially fertile piece of engineering.  In the
abstract it should
	happen.  But I don't think that its existence should gate pNFS,
which (IMHO)
	should exist to manage existing kinds of storage architectures.
	
	Craig

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 17 Nov 2004 13:45:54 -0500

As far as I am concerned, "acting like a normal v4 server" in terms
of the minor versioning applies only to the interaction between the
client and the meta-data server aka the v4 server.  As far as the
interaction between the client and the data servers, I think that
minor versioning rules do not necessarily apply in all cass since 
we may define the storage protocol as being only 4.x for x >= y, 
for whatever y we decide.

-----Original Message-----
From: William A.(Andy) Adamson [mailto:[email-masked]] 
Sent: Wednesday, November 17, 2004 12:54 PM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt 

-->Andy

---

From: "Halevy, Benny" <[email-masked]>
Date: Fri, 19 Nov 2004 10:02:21 -0500

I agree with Craig and I'd like to add a couple of points:

All major SAN FS players were involved with pNFS
from the very early stages.  Doing pNFS as an extension
to NFSv4 was the proposed way to go
from day one (http://www.citi.umich.edu/NEPS/)
and that was discussed and accepted both by the
participants of the ad-hoc pNFS discussion group
and by members and chairmen of the nfsv4 W-G.

I assume that the SAN filesystems folks have
joined this working group if they weren't subscribed
already.

> Personally, I'd argue against trying to merge
> them, on the grounds that that is the job of
> a management layer.  SNIA is going to wrap a
> management layer around parallel access 
> protocols no matter whether there is one big
> one or N little ones.  The time it takes to
> do this will be about the same in either case.
> And since N little IETF protocols can be developed
> in parallel in less time than one big one, that
> argues that you'll get a better time to market
> for the whole package by keeping pNFS small.

I don't see pNFS as "one big one".  The proposed
extensions represent a functional layer
common to several existing SAN and object based
filesystems. At the pNFS level, the gory details
of the underlying filesystem are passed to the
client in embedded, type-coded opaque data structures
(e.g. the layout).  Our intention is to standardize
sub-types within the IETF as separate RFCs that can
(and should) be developed in parallel to the pNFS
extensions.

Spencer Shepler suggested that this W-G should also
work on the server-to-server storage management
protocol which I think is a fine idea, yet I
think that it shouldn't be mandatory to implement,
as other implementation that use different
architectures such as SAN or OSD storage
might find this requirement cumbersome.

 -----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Craig Everhart
Sent: Friday, November 19, 2004 5:32 AM
To: [email-masked]
Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

Alan Yoder <[email-masked]> wrote on 11/16/2004 01:03:27 PM:

> In other venues, protocols that do for block 
> protocols what pNFS does for NFS are being
> worked on.  The folks I've talked to call
> these "SAN Filesystems".  

Right.

> If anyone wants these efforts to be merged, 
> I'd think they ought to be talking to 
> the SAN Filesystem people too, not just the 
> folks here.

Who else needs to be told?

My day job is working on a SAN file system, so at least
some of the SAN Filesystem people are here.

> Personally, I'd argue against trying to merge
> them, on the grounds that that is the job of
> a management layer.  SNIA is going to wrap a
> management layer around parallel access 
> protocols no matter whether there is one big
> one or N little ones.  The time it takes to
> do this will be about the same in either case.
> And since N little IETF protocols can be developed
> in parallel in less time than one big one, that
> argues that you'll get a better time to market
> for the whole package by keeping pNFS small.
> 
> Alan

Are you saying that nfsv4 isn't the right group to
be discussing this, that it's not properly an IETF
thing?

I don't believe that the pNFS effort has imagined that there
would be one big protocol, but rather a smaller one with
a choice of third-party data access protocols.  Perhaps, as
discussion has happened here, use of one of those protocols would
be required for conformance.  I don't really say it that
way; here's why.  The pNFS effort was doing a couple of
things: one of them was to define new third-party data
transports as NFS extensions, but equally (or, from my
own point of view, more so) it was trying to integrate
third-party transport mechanisms into NFS, allowing
NFS to be the client-server protocol that could be used
as part of a SAN Filesystem (as you name it).  I'm
myself very interested in an NFS client being able to
do block SCSI over FC to a storage device.  But the particular
protocol that could be used to talk to a storage device is
more likely to be chosen in advance by whoever deploys the
storage device.  It could be that a single machine could
talk block SCSI over FC to one device, OBS SCSI to another
device, and one or two variants of sub-file that we see
being discussed on the list to a third or fourth device.
But it may not be likely that *all* those protocols will
be deployed at a given client, and it's not likely that their
definitions and interoperability stories will mature at the
same rate.  So what we had discussed as a possibility was
thinking of the pNFS extension to NFS as a single entity with
multiple sub-types.  The sub-types could progress towards
interoperability status independently, while at least one
of them would be required in order to allow the production
of "working code" for pNFS.  That is, pNFS proper, dealing
with layouts and such, doesn't care which sub-type is in use,
as long as there is one that is implemented by the particular
client and the particular third-party storage (storage system
or OBS disk or "data server" via NFS).

So some folks are saying that the best way to achieve this
is to pick one sub-type and consider that alone for pNFS first,
then strive to add other sub-types only after pNFS is much
further along.  But doing this risks consensus over the choice
of one of the variants, and it's also a recipe for
substantial delay for the less-preferred sub-types.

Would it be unprecedented to have a handful of possible
sub-types and require one or more of them, but no matter which,
to test pNFS implementations?  I remember the early IP
specification RFC, followed by a few on "IP on Experimental
Ethernet" and "IP on Chaosnet" kinds of stacking discussions,
but they may not be that helpful as models--we're not really
doing stacking, but we have a variety of embeddings for the
protocol.

---

From: "Halevy, Benny" <[email-masked]>
Date: Fri, 19 Nov 2004 10:10:29 -0500

Julo wrote:
> Using IPsec is an expedient way to have a secure channel protect 
> the integrity (not necesaricly the privacy) of everything transmitted.
However Object Storage may use 
> other techniques (signed commands and/or commands and data when IPsec is
not deployed or deemed too expensive (BTW IPSec is the "least expensive"
security to implement if you ignore key deplyment intricacies and has the
widest hardware support (critical a high speed). 

I think that what Mike was saying is that GSS-API could be used
to transfer the capabilities in a secure and private way
without requiring IPSec.  I like that.

Benny

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Julian Satran
Sent: Friday, November 19, 2004 4:33 PM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

> specifically, RPCSEC_GSS. The i-d needs to justify its direction.

The Object store capability protects the storage from unathorized access
without having to mainatin 
duplicate ACL at each storage device. This is substantially different than
the service provided by RPCSEC_GSS. 

> GSS-API would be the ideal way to wrap an HMAC.
Using IPsec is an expedient way to have a secure channel protect 
the integrity (not necesaricly the privacy) of everything transmitted.
However Object Storage may use 
other techniques (signed commands and/or commands and data when IPsec is not
deployed or deemed too expensive (BTW IPSec is the "least expensive"
security to implement if you ignore key deplyment intricacies and has the
widest hardware support (critical a high speed). 

> >   the client and metadata server as between the client and data server.
injects
protocol
conference,

---

From: "Yoder, Alan" <[email-masked]>
Date: Fri, 19 Nov 2004 07:00:21 -0800

> Sent: Thursday, November 18, 2004 7:32 PM
> To: [email-masked]

> some of the SAN Filesystem people are here.

My apologies.  I'm on an internal reflector, which
means my messages have to be approved by the 
moderator.  This message was written several days 
ago just before it became obvious that SAN filesystem 
people are very much involved.

Alan

===============================================================
Alan G. Yoder                                    [email-masked]
Technical Staff                          
Network Appliance, Inc.                            408-822-6919
===============================================================

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 29 Nov 2004 10:35:49 -0500

Dean Hildebrand:
> [email-masked] wrote:
> > We have a lot of issues to decide about what needs to be mandatory in
> > order to prevent pnfs from Balkanizing but I don't think that it should
> > be an open question whether a pnfs server has to inter-operate
> > meaningfully with a v4.0 client.  I'm sure there will be arguments about
> > "meaningfully" but getting through the Connectathon tests is a very low
> > minimum bar.
>
> Would the NFSv4.X server have to survive Connectathon alone? 

It depends on the form of help that you are thinking about.  It is clear that
a form of help that the client doesn't actually see is allowed.

For example, if the meta-data server is unable to perform a READ or WRITE
operation on its own (disk connectivity issues) it might have a helper server
that does have connectivity to the data disks (just as the clients do).  If
the READ or WRITE is forwarded to that helper server and performed there for
the meta-data server who then responds upon being notified of completion by 
the helper (it is possible for the helper to directly respond to the clients 
although handling of delegations and share reservations require extra care 
when there are potential outstanding IO operations on the helper).  

> It seems
> that if pNFS extensions allowed a pNFS server, in conjunction with a set of
> data servers, to pass Connectathon then that is sufficient.  

No, that is not sufficient.  It is necessary, however.  Clearly, a pnfs
client (i.e. a v4.x client with support for the pnfs extensions) and a
pnfs server (i.e. a v4.x server with support for the pnfs extensions)
needs to be able to Connectathon, or else the extensions are woefully
inadequate.

However, the server does have to able to support v4.0 clients (performing
the Connectathon test or whatever) and v4.x clients that do not support
the pnfs extensions.  Since the requirement is to support clients that do
not support extensions (e.g. v4.0 clients), then any extensions you come up 
with can't help you help you in this regard.

> This could be
> thought of an extension of where a NFSv4.X server uses FS_LOCATIONS to
> redirect all clients to other servers, thereby not passing Connectathon by
> itself but can "get by with a little help from its friends".

No extensions.  You need to use something which is in v4.0.

Since FS_LOCATIONS and the MOVED error are in v4.0, you might argue that a
referral event (in the case of v4.0) or a migration event (in the case of a
v4.x client without pnfs support), might provide sufficient support to stay
within the letter of the law, but I think attempting to try to evade/avoid
the requirement to implement v4 as it stands is the wrong way to go for
lots of reasons.

First, even if the referral/migration approach is judged conformant, making it 
actually work is no picnic.  Share reservations, delegations, and byte-range
locks are going to be performed on two different servers when you do things
this way.  They would have to communicate so that conflicting locks are not 
given out.  Dealing with reboots of either the meta-data server or the helper,
transient connectivity loss between the two etc., is doable 

The simplest way of conforming to the spec's requirement that you support
existing ops (e.g. READ and WRITE), is simply for the meta-server to 
implement those ops, rather than search for ways of getting by without
doing that.  By hypothesis, we have server which has sufficient information 
that it can allow a client to perform the operation in question.  It is hard 
to believe that there is any overwhelming difficulty in taking that same 
information and  simply reading or writing the data, as requested by the 
client.  If there is no direct connectivity to the storage in question, 
there must be indirect connectivity, since it can talk to the clients who 
must have that connectivity.  One or more helper servers with the same 
connectivity as clients must be able to perform the work.

Apart from any spec issues, as a practical matter, people are going to have to 
support v2 and v3 clients as well.  Also, again apart from how you parse 
the spec, people are going to want support for existing v4 clients that 
don't have referral/migration support (i.e. all of the ones that exist today).  
Further, a server that tries to force a major environmental change such as 
migration whenever a client decides that for a given IO, the best choice is 
simply to issue a READ or WRITE, is a very poor implementation choice, even 
if legal.  NFSv4 has READ and WRITE ops and the fact that pnfs will be a set 
of v4 extensions pretty much means that if you want to be a pnfs server you 
need to figure out how to implement READ's and WRITE's and by implement READ's 
and WRITE's, I mean get the data from the file and write the data to the file.