Subject: RE: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 16 Nov 2004 13:14:46 -0500

> > Note however that nobody has said that requiring pNFS servers to
> > support v4 as a storage protocol is an OK to thing to do.  I
> > certainly don't think it is and I suspect Brent agrees.  If
> > Nico also agrees then maybe we can move on.  Otherwise, we can
> > have a detailed discussion on this issue.
> 
> It's a package deal to me.  NFSv4 must be the REQUIRED to implement
> protocol for client<->storage and client<->meta-data.

So the question is, how do you create a meaningful requirement?  
OK, it's mandatory to provide a NFSv4-style layout.  Nothing could
be simpler.  The layout for file handle FH says that the data is
at FH' and IP address X (one of the servers IP addresses) and your
done.  Is that OK?  The performance of accessing the data using 
that layout will be no better than accessing it over standard 
NFSv4, but do we want to get in the business of making performance
requirements?  You can argue that this is an evasion of the
requirment but it just isn't possible to draw the line between
evasion and avoidance in a way that sticks.  So a client would
learn to ignore this sort of layout, and use the ones that that
server makes faster.  In which case, why should we ask the server
to pretend?  If he is set up so that pnfs-blocks is fast, then 
why should we ask him to tell the client that pnfs-files is 
available, when it isn't, for all practical purposes.

> > The reason that we can't make this out of scope is that the proposal
> > which is under discussion is intended to allow block protocols.  If
> > block protocols were not possible, because of valid security issues,
> > then the base protocol would look different.  So, even though we 
>                                                 ^^^^^^^^^^^^^^^^^^
> > can discuss such security issues, the bar for those raising these
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > security objections is very high.  I think would have to show that no
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > possible block protocol could be acceptable (or at least worthwhile), 
> > which is a very hard thing to show (since it isn't true or at least 
> > I don't think it is).

> Hooey.  The NFSv4 WG charter says nothing about pNFS, so I think all
> aspects of any pNFS proposals are up for discussion until consensus is
> reached.

Sure, but your right to discuss things has to deal with my (and others')
right to decide what arguments to listen to.  What I guess I was saying
in the above sentence is that for me, the appropriate time to discuss
the security flaws of a particular proposal to use a blocks protocol
is when such a proposal is actually made, i.e. there is a document that 
really defines what is proposed.  Until then, we are dealing with 
stuff that is speculative, unless you can show me that it is impossible
that any proposal in this vein will be acceptable.   

> Anyways, the burden is on the authors.  Let them show us the various
> combinations they have in mind and let them tell us what their security
> and operational properties are.

I agree, once they want to make specific proposals on additional storage
protocols.  For me, saying that objects are interesting and that the
general pnfs framework should accommodate blocks protocols, does not 
count as making a specific proposal.

> Then we can discuss this and reach a consensus on what security
> properties we _want_ and then we can talk about what's in scope and
> what's not.

It's just a simple matter of reaching consensus :-)  Sigh.

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]]
Sent: Tuesday, November 16, 2004 11:45 AM
To: Noveck, Dave
Cc: Brent Welch; Mike Eisler; [email-masked]
Subject: Re: [nfsv4] comments on draft-welch-pnfs-ops-00.txt

On Tue, Nov 16, 2004 at 10:32:27AM -0500, Noveck, Dave wrote:
> So it appears that Nico and Brent are in violent agreement that
> requiring clients implementing the pNFS extensions to support NFSv4 
> as a storage protocol is an OK thing to do.  I agree, although
> I will note that requiring it will make no difference, since all
> clients are going to do that anyway (if you a v4.x client and you
> have general pnfs client logic, you would have to be crazy not to
> support v4.x as a storage protocol).  But anyway, maybe we have 
> a consensus on this issue and can move on.
> 
> Note however that nobody has said that requiring pNFS servers to
> support v4 as a storage protocol is an OK to thing to do.  I
> certainly don't think it is and I suspect Brent agrees.  If
> Nico also agrees then maybe we can move on.  Otherwise, we can
> have a detailed discussion on this issue.

It's a package deal to me.  NFSv4 must be the REQUIRED to implement
protocol for client<->storage and client<->meta-data.

> As far as Brent saying:
> 
> > I would like to table all discussion of block protocol security
> > as being out of scope.  I think it clouds the issue and gets in
> > the way of working out an acceptable pNFS extension.
> 
> It certainly is tempting, but I don't think we can say, in Lenin's
> words, "Comrades, this is no time to have an opposition".
> 
> The reason that we can't make this out of scope is that the proposal
> which is under discussion is intended to allow block protocols.  If
> block protocols were not possible, because of valid security issues,
> then the base protocol would look different.  So, even though we 
                                                ^^^^^^^^^^^^^^^^^^
> can discuss such security issues, the bar for those raising these
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> security objections is very high.  I think would have to show that no
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> possible block protocol could be acceptable (or at least worthwhile), 
> which is a very hard thing to show (since it isn't true or at least 
> I don't think it is).

Hooey.  The NFSv4 WG charter says nothing about pNFS, so I think all
aspects of any pNFS proposals are up for discussion until consensus is
reached.

Anyways, the burden is on the authors.  Let them show us the various
combinations they have in mind and let them tell us what their security
and operational properties are.

Then we can discuss this and reach a consensus on what security
properties we _want_ and then we can talk about what's in scope and
what's not.

> Nico writes:
> > So what's in scope is: what sort of protocol should pNFS clients use to
> > talk to the storage.
> 
> Protocol or protocols?  If it is one, the only possible answer is
> NFSv4 and I think everybody agrees that that is a member of the set 
> of storage protocols and in fact the first one that we should be
> working on.  Any dissent?
> 
> So I think the question is best put as:
> 
>    What sort of protocols may pNFS clients use to talk to the storage?
> 
> And allowing some scope for individual choice within that set.

Ok.  See above.

[...]
> When Nico writes:
> 
> > Perhaps this will fly past the IESG, perhaps not.  I think we should
> > inquire with them at some point (and their answer may be that they can't
> > yet answer so you just have to try to deliver a document to them and
> > see).  Of course, first we need WG consensus on this.
> 
> I think we need to figure out how we can make progress without necessarily
> have a consensus on the general goodness of block protocols for pnfs
> because we may never get it.  Brent has talked about the optionality 
> of additional storage protocols and my understanding is that he is
> talking about the fact that implementers may choose to implement them
> (or not) and that, if implemented, users may choose to use them (or
> not).  The protocol development process needs to have a form of 
> optionality which I'll call "severability" to distinguish it.  We need:
> 
> (1) A general pnfs architecture that allows a set of storage protocols.
> 
> (2) A description of how v4.x would be used as a storage protocol,
>     including security considerations.
> 
> (3-n) A description of how the additional storage protocols would work
>       in the pnfs framework, including security considerations, with one
>       document for each protocol.
> 
> For me, (1) and (2) are the highest priority items and (1) is required
> for everybody.  I'm assuming that there will be a v4.x spec that covers
> (1) and (2) and that there may have to be negotiation with the IESG
> as to (3), (4), etc. as those documents are produced but we have to be 
> clear that that (1) and (2) can go ahead while hashing out the issues 
> associated with (3-n) to the IESG's satisfaction.  The explanation in
> (1) will have to explain that the intention is to allow block protocols
> if and only if the security issues can be resolved.  I take the laissez-
> faire position that their optionality constitutes an adequate resolution,
> but I am worried that the IESG will adopt a more dirigiste approach.  I
> think pnfs as a whole has to proceed regardless of that (I fear) endless
> debate.    

Yes.

Thanks,

Nico
-- 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 Nov 2004 12:30:58 -0600

On Tue, Nov 16, 2004 at 01:14:46PM -0500, Noveck, Dave wrote:
> requirements?

Huh?  Why couldn't the meta-data server say

[{fs_location1, FH', byte-ranges}, {fs_location2, FH', byte-ranges}, ..,
 {fs_locationN, FH', byte-ranges}]

And if OSD-like extensions to NFSv4 are needed that's fine with me.

>                You can argue that this is an evasion of the

See above.  I do think that NFSv4 may need extending to make it a
high-performance proposition for this.

> that any proposal in this vein will be acceptable.   

The authors termed the security of pNFS "comparable" to NFSv4's -- in
their slides at last week's meeting.  No, I have not read their
document, but this is enough for me to get concerned.  Here I start to
repeat myself; I'll not and instead I offer to review the new version of
the pNFS I-D(s).

> count as making a specific proposal.

I will now shut up and wait for the -01.  We're close to going in
circles.

The authors now have several copies of my comments on what I'd like to
see from them, as well as lots of comments from others, including
Mike's, who started the whole pNFS security discussion.

Over to the authors.

Cheers,

Nico
--

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Tue, 16 Nov 2004 14:56:47 -0500

>  {fs_locationN, FH', byte-ranges}]

an fs_locations-like format with server and pathname is one way to express a 
file layout that allows striping of a file over NFSv4 data servers. this means 
that the pNFS client is doing an OPEN (pathname) to each data server in order 
to establish clientid, stateid, etc. so, why the FH argument in the layout? 
the OPEN will return the data servers idea of FH and the stateid.

there is the question of a storage management protocol between the NFSv4 
metadata server and the NFSv4 data servers - as dave pointed out, a SETATTR 
with ACLs needs to be propagated to the data servers. the namespace will need 
to be kept in sync -  RENAME, REMOVE, CREATE, will also need to be 
synchronized between metadata and data servers.

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 Nov 2004 13:59:05 -0600

> the OPEN will return the data servers idea of FH and the stateid.

Details.

> synchronized between metadata and data servers.

Synchronization can't be avoided with iSCSI either, not if the iSCSI
servers are serving different-but-mirrored/RAIDed storage.

Nico
--

---

From: Craig Everhart <[email-masked]>
Date: Tue, 16 Nov 2004 16:02:01 -0500

My IBM disclaimer: IBM may have intellectual property in this area.
(It's been a while since I posted to nfsv4.)

"William A.(Andy) Adamson" <[email-masked]> wrote on 11/16/2004
02:56:47 PM:
> that the pNFS client is doing an OPEN (pathname) to each data serverin
order
> the OPEN will return the data servers idea of FH and the stateid.

Now, it's just my understanding, but I think that the motivating example
was
one in which the data server didn't know anything about the directory
structure.  In that case, you'd need to get the correct FH from the
metadata
server that would be usable for things like read and write to the data
servers.

Admittedly, you could also do things so that the data server would require
an OPEN, and would thus share pathnames (and, presumably, directories) with
the metadata server.

> synchronized between metadata and data servers.

Certainly the namespace-changers don't need to be synchronized unless
the namespace itself is mirrored, which isn't necessarily a requirement.
And isn't the requirement not that a SETATTR(acls) be reflected in the data
servers, but rather that equivalent security to that be maintained, whether
by a capability mechanism or some other strong mechanism?  I'm assuming
that
capabilities would have to be revoked, or be able to be revoked, on the
occasion of a SETATTR(acls), but not that the new ACLs themselves would
necessarily have to be sent to the data servers.

            Craig

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 16 Nov 2004 17:13:40 -0500

>  {fs_locationN, FH', byte-ranges}]

Yech. That is potentially an *unbounded* list.

Cheers,
  Trond

-- 
Trond Myklebust <[email-masked]>

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 16 Nov 2004 16:17:43 -0600

> Yech. That is potentially an *unbounded* list.

No, it's not.  It's clearly bounded *and* this is exactly the sort of
thing that was presented last week at the meeting, only there the
emphasis was on blocks.

Nico
--

---

From: Marc Eshel <[email-masked]>
Date: Tue, 16 Nov 2004 15:17:24 -0800

Nicolas Williams <[email-masked]> wrote on 11/16/2004 02:17:43

> Nico

This is great we get file level referral with {fs_location, FH',
byte-ranges( 0 - infinity )}
which is good among other things for load balancing.

Marc.

---

From: Brent Welch <[email-masked]>
Date: Tue, 16 Nov 2004 17:18:38 -0800

>>>Nicolas Williams said:
 
 > thing that was presented last week at the meeting, only there the

I can't help but rephrase this as "we got side-tracked on blocks
and I never got a chance to describe how cool object storage is :-)

I appreciate everyone's feedback on this.  I think a concrete proposal
for both the NFSv4 storage protocol and object storage protocol
will provide a useful basis for further discussion on the pNFS 
umbrella.

--
Brent Welch
Software Architect, Panasas Inc
Delivering the premier storage system for scalable Linux clusters

www.panasas.com
[email-masked]

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 17 Nov 2004 00:44:59 -0600

> will provide a useful basis for further discussion on the pNFS 

Yes, we need to compare the options.

---

From: Julian Satran <[email-masked]>
Date: Wed, 17 Nov 2004 09:27:36 +0200

> and I never got a chance to describe how cool object storage is :-)

It goes without saying that I agree.

> will provide a useful basis for further discussion on the pNFS 

That would be good provided that we don't go on the path of replicated 
ACLs.
Although appealing - as it pushes aside the access control issue it 
creates deployment nightmare.