Subject: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 2 Oct 2012 13:31:50 -0400 (EDT)

Hi Benny,

As we presented to the working group in July (IETF 84), in our work implementing pNFS access to data in the Ceph storage system, our team discovered a need for striping flexibility, similar to that provided by the object layout—in a configuration using the NFS protocol for access to data servers.

We're very interested in the NFS protocol data access concept presented in draft-bhalevy-nfs-obj-00.txt.  We find that the PNFS_OBJ_NFS concept addresses many of the problems we have encountered using the files layout, in particular, it would allow us to use the objects striping model while still using the NFS protocol for transport.

One issue we believe we've found, relates to the proposed credential mapping.  For our use, which does not depend on using ordinary NFS servers as data servers, the mandated replacement of the client's RPCSEC_GSS credential would be problematic for us.

We would like to discuss whether a hybridized model could be acceptable, decoupling the choice of authorization model, from that of the data access protocol.  Then, in addition to the currently proposed behavior, the MDS could direct the client to use its ordinary RPC credential set for operations on the data server.  Clearly, the MDS would be indicating by selecting this mode of operation that it would provide its own (e.g., out-of-band) fencing support.

Thanks,
Matt

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 2 Oct 2012 22:36:10 +0000

We will have the right people in the room in Atlanta to further this discussion (if it doesn't close before then)?

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Matt W. Benjamin
Sent: Tuesday, October 2, 2012 10:32 AM
To: nfsv4 list
Subject: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt

Hi Benny,

Thanks,
Matt

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 2 Oct 2012 21:26:25 -0400 (EDT)

Is (are) the WG meetings scheduled?  Which day(s)?

----- "Spencer Shepler" <[email-masked]> wrote:

> We will have the right people in the room in Atlanta to further this
> discussion (if it doesn't close before then)?
> 

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 3 Oct 2012 01:27:17 +0000

Draft schedule is due out on Thursday of this week.

-----Original Message-----
From: Matt W. Benjamin [mailto:[email-masked]] 
Sent: Tuesday, October 2, 2012 6:26 PM
To: Spencer Shepler
Cc: nfsv4 list
Subject: Re: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: Benny Halevy <[email-masked]>
Date: Thu, 4 Oct 2012 09:15:29 +0200

Yes, continuing the discussion in Atlanta will be perfect.

Benny

On Wed, Oct 3, 2012 at 12:36 AM, Spencer Shepler <[email-masked]>wrote:

> As we presented to the working group in July (IETF 84), in our work
> implementing pNFS access to data in the Ceph storage system, our team
> discovered a need for striping flexibility, similar to that provided by the
> object layout—in a configuration using the NFS protocol for access to data
> servers.
>
> We're very interested in the NFS protocol data access concept presented in
> draft-bhalevy-nfs-obj-00.txt.  We find that the PNFS_OBJ_NFS concept
> addresses many of the problems we have encountered using the files layout,
> in particular, it would allow us to use the objects striping model while
> still using the NFS protocol for transport.
>
> One issue we believe we've found, relates to the proposed credential
> mapping.  For our use, which does not depend on using ordinary NFS servers
> as data servers, the mandated replacement of the client's RPCSEC_GSS
> credential would be problematic for us.
>
> We would like to discuss whether a hybridized model could be acceptable,
> decoupling the choice of authorization model, from that of the data access
> protocol.  Then, in addition to the currently proposed behavior, the MDS
> could direct the client to use its ordinary RPC credential set for
> operations on the data server.  Clearly, the MDS would be indicating by
> selecting this mode of operation that it would provide its own (e.g.,
> out-of-band) fencing support.
>
> Thanks,
> Matt
>
> --
> Matt Benjamin
> The Linux Box

---

From: Benny Halevy <[email-masked]>
Date: Thu, 4 Oct 2012 09:29:43 +0200

> object layout—in a configuration using the NFS protocol for access to data

Great.  I'm glad it may be useful for your use case as well!

> credential would be problematic for us.

That's for data server exporting NFSv4 or higher.

> protocol.

Interesting idea.  I wonder though if that doesn't belong to client
configuration choice.
For example, we don't propose a way (on purpose) for the server to convey
recommended
mount options to the client to control, e.g. [rw]size, as this setting
might be different for different
clients, on different OSs, or over different network configurations.

> Then, in addition to the currently proposed behavior, the MDS could direct
> the client to use its ordinary RPC credential set for operations on the
> data server.  Clearly, the MDS would be indicating by selecting this mode
> of operation that it would provide its own (e.g., out-of-band) fencing
> support.
>

Can you please write up proposed text and XDR for this feature?

Benny

---

From: "Adam C. Emerson" <[email-masked]>
Date: Thu, 04 Oct 2012 12:42:20 -0400

Good morning, Mr. Halevy,

At Thu, 4 Oct 2012 09:29:43 +0200,
> On Tue, Oct 2, 2012 at 7:31 PM, Matt W. Benjamin <[email-masked]> wrote:
[snip]
>     We would like to discuss whether a hybridized model could be acceptable, decoupling the
>     choice of authorization model, from that of the data access protocol.
> 
> Interesting idea.  I wonder though if that doesn't belong to client configuration choice.
> For example, we don't propose a way (on purpose) for the server to convey recommended
> mount options to the client to control, e.g. [rw]size, as this setting might be different for
> different

I don't believe that 'client choice' makes sense in this context.
Since the data server, on granting a layout, provides the client with
both permission and means to access a the file data directly, I
believe it makes more sense for the server to be able to dictate the
kind of authorization that the client would use.  In our case, we
would like the server to instruct the client to authenticate itself to
the data server exactly as it would to the meta-data server.

We also believe that an additional flag, instructing the client to
restrict itself to the data-server subset of NFSv4.1 (session
operations, secinfo, read, write, and commit) when speaking to the data
server would be useful, especially in cases where a data server had
some MDS functionality, but would have to call out to satisfy metadata
requests on the file in question.

Respectfully yours,

Adam C. Emerson                                  <[email-masked]>

---

From: Benny Halevy <[email-masked]>
Date: Tue, 09 Oct 2012 14:05:42 +0200

On 2012-10-04 18:42, Adam C. Emerson wrote:
> kind of authorization that the client would use.  

I'm inclined to agree as the MDS is responsible for making the security policy
decisions and the DS is responsible for enforcing the decisions.

> In our case, we

Are you requesting for a simple boolean flag to represent that?

> requests on the file in question.

What does this buy you?
The client has no business doing other operations on the DS in any case.
If you control the DS and disallow other operations the client can
try them if it wants to and it will fail.

Benny

---

From: "Adam C. Emerson" <[email-masked]>
Date: Tue, 09 Oct 2012 11:30:51 -0400

Mr. Halevy,

At Tue, 09 Oct 2012 14:05:42 +0200,
> Are you requesting for a simple boolean flag to represent that?

I think a boolean flag should be enough, though making the onc_auth a
switched union might be cleaner.

> try them if it wants to and it will fail.

The client has to send EXCHANGE_ID wth EXCHGID_FLAG_USE_PNFS_DS set
before any other operations, so it doesn't really have an opportunity
to try other operations before establishing the session.

Also, in the case of a server that isn't aware that it's participating
in pNFS (the use case you describe in 13.5 where the MDS changes
permissions on the server) it's not clear to me that the client
SHOULDN'T try to OPEN the file, which it ought never do on a
cooperating pNFS DS.

Respectfully yours,

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 9 Oct 2012 15:48:51 +0000

> SHOULDN'T try to OPEN the file, which it ought never do on a

Adam,

This is what I mean about having crisper definitions of "server".

The way I read what you wrote, the MDS fences off a file on the
DS by modifying the permissions on it. This implies the MDS
has some mapping from its namespace to that of the DS.

And while that makes more sense to me than the MDS modifying 
just its namespace, I still struggle with it.

Say two clients are accessing a file and they both have read locks.
Client A decides to promote up to a write lock. How are the permissions
modified on the DS such that only client A has access?

Thanks,
Tom

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 9 Oct 2012 15:52:56 +0000

> SHOULDN'T try to OPEN the file, which it ought never do on a

Alright, so what does all this extra complexity and extra client-server
back and forth interaction buy us?

Where's the killer app that cannot be satisfied with today's pNFS
technologies?

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Boaz Harrosh <[email-masked]>
Date: Tue, 9 Oct 2012 17:42:15 -0700

> modified on the DS such that only client A has access?

This is done with the pnfs-objects type of fencing. All clients are
fenced with error-caps-changed (In OSD it is caps version changed)
and they all return their current layout and ask for a new one, which
will now include the new permissions. In your example above client B
will nor receive a new layout. In objects it happens all the time,
every truncate, attribute changes, and locks will recall all layouts
and issue new ones, not only fencing operations.

In regard to a DS ability to actually force clients adhering to the
permissions received by the MDS, Say client B receives a read-only
layout yet tries to write to the file. Well that is version dependent.
With NFSv3 NFSv4 access to DS it is not enforced. MDS will need to trust
it's clients. But with NFSv4.1 we can pass a stateid / filehandle to DS
that will enable the DS to do so.

Note how in NFSv4.1 a DS-Filehandle is a very new animal, because it is
an upto 128 bytes opaque that is not at all constrained by the NFS2-4.1
filehandles rules. Most specifically it need not be persistent or even
the same, for the same file. All it is, is a "something" that is sent in a
layout and is then received by the DS on write/read.

Actually with a back-channel protocol even an NFSv3-4.0 DS can preform
all these checks, by the MDS to DS utilizing some protocol just like
OSD does, and passed on the "DS-filehandle". Actually nfs fhandle is bigger
then OSD caps.

So basically implementation changes depending on if there is a back-channel
protocol and/or control of the underline DS software by the MDS. Benny's
motivation is that it will be possible to not have one and have enough numbs
to be able to operate. And he understands and even states so, that without
a back-channel protocol like in NFSv3 case the client must be trusted.

Cheers
Boaz

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 10 Oct 2012 02:19:59 +0000

> that will enable the DS to do so.

How does the MDS inform the DS that the stateid is for writing only?

> layout and is then received by the DS on write/read.

Actually an argument against this proposal - the DS filehandle only makes
sense in context of a layout and a stateid. With a NFSv3 server, the client
can both cache and use this DS filehandle forever.

> OSD does, and passed on the "DS-filehandle". Actually nfs fhandle is bigger

But there is no requirement to have a Control Protocol.

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 10:54:21 -0700

> How does the MDS inform the DS that the stateid is for writing only?

Same as files-layout, today, as a back-channel private protocol of sorts.

Again, if MDS and all DSs belong to a controlled software version,
and some protocol is established between them then they can enforce
granted access by MDS. Else only an honest client will work properly
and will be fenced if needed. A malicious client could fake credentials
and most probably will be able to sneak in. But this is a none complaint
client and is part of the Admin's problem.

Though even in the case of a sneaky client with benny's plan we can make
it very hard for him. Because what happens is that now a client has received
a filehandle from MDS. (He cannot retrieve a filehandle by himself because
the directory access is set in such a way that clients do not have readdir
or lookup permission, hell they don't even have a path). The uid/gid he
received only allow reads. There are in the system some other uid/gid of client
A which has write permissions. But can client B actually know what are those
other uid/gid? You tell me, In NFSv3 can I block a client from retrieving the
permitted uid/gids of a file. I'm sure you can with NFSv4 with ACLs right.
So the sneaky client will have to probably guess them, good luck with that.

[In NFSv3, I'm sure that if the backing FS has POSIX ACLs support one can
 give out gids such that, the client will never know of these gids but will
 grant access to the file]

So you see with controlled access credentials received in layout any NFS
server should be pretty safe.

And with any "controlled" nfs server that is also part of the same
software distribution, with a backchannel, you can make it 100%
full proof.

Cheers
Boaz

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 10 Oct 2012 19:28:08 +0000

> Same as files-layout, today, as a back-channel private protocol of sorts.

A back channel is not mandated in the proposal.

> granted access by MDS.

I don't disagree with that. I disagree with the fact that the proposal does
not currently provide for it.

And if the use case, which has yet to be presented, is for Vendor A
to reuse Vendor B's NFS server as a backend AND Vendor B's server
is not Open Source, how do they use this private, unpublished protocol,
which is not a standard?

> Else only an honest client will work properly
> or lookup permission, hell they don't even have a path).

Where is that detail in the proposal?

> The uid/gid he
> permitted uid/gids of a file.

Without having access to the source code for the NFS server?

Again, the proposal as it stands applies for NFS servers which may
be part of some unified back end, but it also applies for any NFS server.

> I'm sure you can with NFSv4 with ACLs right.

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 13:48:46 -0700

On 10/10/2012 12:28 PM, Haynes, Tom wrote:
<>
> be part of some unified back end, but it also applies for any NFS server.

Without any source code.

Say the MDS creates a file on DS_A with uid=100 and gid=100 and bits
are set as user=W+ and group=W-. I now give client A uid=100/gid=100
and client A can write. But I give client B uid=101/gid=100 so he
can only read, no?

on fencing off MDS changes files creds to uid=102/gid=102 and both
clients are fenced off and need to return the layout and get a new
one.

These uid/gid above are those received in the layout and the client
uses them in DS communication. They must not be confused with
file's real uid/gid on the MDS and those that the client holds
for the file. And/or those he used in MDS open.

My point was that for extra security if the underline system has
POSIX Acls, or the server is NFSv4 then the allowed groups can
even be unpublished and the sneaky client cannot know what creds
to fake.

With a standard compliant client we will never have a problem.

Cheers
Boaz

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 10 Oct 2012 21:51:18 +0000

My uid is 1066, gid = 30. I want to access the same filesystem from client A
and client B. I'm just a simple user who has no clue that I am using the
newest OSD pNFS layout.

My friend Mikey has uid 1067, gid 30, had it for years. I don't want him looking
at my fantasy football league notes - he cheats. But I do have to share
group code with him.

How does you scheme work?

For every trick, there is a corporate IT requirement it will violate. :->

BTW - look at 5661's Section 12.2.4

> 12.2.4.  Storage Device
> 
>    A storage device stores a regular file's data, but leaves metadata
>    management to the metadata server.  A storage device could be another
>    NFSv4.1 server, an object-based storage device (OSD), a block device
>    accessed over a System Area Network (SAN, e.g., either FiberChannel
>    or iSCSI SAN), or some other entity.

And then the definition of Control Protocol in 12.2.6:

> 12.2.6.  Control Protocol
> 
>    As noted in Figure 1, the control protocol is used by the exported
>    file system between the metadata server and storage devices.
>    Specification of such protocols is outside the scope of the NFSv4.1
>    protocol.  Such control protocols would be used to control activities
>    such as the allocation and deallocation of storage, the management of
>    state required by the storage devices to perform client access
>    control, and, depending on the storage protocol, the enforcement of
>    authentication and authorization so that restrictions that would be
>    enforced by the metadata server are also enforced by the storage
>    device.
> 
>    A particular control protocol is not REQUIRED by NFSv4.1 but
>    requirements are placed on the control protocol for maintaining
>    attributes like modify time, the change attribute, and the end-of-
>    file (EOF) position.  Note that if pNFS is layered over a clustered,
>    parallel file system (e.g., PVFS [51]), the mechanisms that enable
>    clustering and parallelism in that file system can be considered the
>    control protocol.
> 
> 12.2.7.  Layout Types

I read that as stating the Control Protocol requirement is outside the
Storage Device and Layout Type. I.e., block-based and object-based
layouts have these same requirements on not just file access, but
modify time, change attribute, and EOF position.

I like the concept of Stone Soup computing (see
http://en.wikipedia.org/wiki/Stone_Soupercomputer), but I also
believe we can not abandon these requirements just to use some
old gear.

Where is the use case?

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 10 Oct 2012 22:00:52 +0000

> believe we can not abandon these requirements just to use some

I don't see that we would need to. It has been quite a few years since I
saw a NAS unit that didn't also support at least one block protocol such
as iSCSI (or even FCP). I'm certainly not aware of a NAS that supports
NFSv4, but not iSCSI.

> Where is the use case?

...and why can you not use a blocks based layout for said use case?

NetApp

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 15:35:16 -0700

> How does you scheme work?

What? are you joking or serious?

Did you not read the draft. (Well I haven't actually ;-) not the last version)

Nothing changes for the user, it is all well hidden in the secret protocol
between client-to-MDS and client-to-DS.

I specifically said:
>> for the file. And/or those client used in MDS open.

> For every trick, there is a corporate IT requirement it will violate. :->
Nope no tricks no violations
> modify time, change attribute, and EOF position.

Sorry sir But I have not violated any of the above sections,
"what so ever", they all stand firm.

Please note that T10-OSD is special in above regard of the "undefined
Control Protocol". Since with T10-OSD that Control Protocol is defined,
it is the same T10-OSD.

But I guess a "defined" Protocol is just a special case of "any undefined
protocol"

> believe we can not abandon these requirements just to use some

Again nothing is violated. Modify time, change attribute, EOF position,
file ownership client-to-server and application-to-file access control
are still working perfectly as before, with all three flavors of
authentication.

The only new proposed thing is that, client-DS communication can
happen with an RPC-NFS wire protocol instead of an T10-OSD.

> Where is the use case?

Existing and new Topologies, old gear, no violation of any
standard, no new software stacks, only few enhancements to
existing opaque buffers, and enhanced behavior of client.

I'm not going to repeat myself anymore. It's my second time
around saying the same arguments, That's it I'm done. Bye

Shalom
Boaz

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 15:46:43 -0700

> as iSCSI (or even FCP). I'm certainly not aware of a NAS that supports

Not relevant, CEPH OSDs and Luster OSSs do not speak iscsi and it makes
no sense. iscsi LUNs are not objects, objects are files with simple name
and no meta-data, they are not at all flat LUNs. LUNs are static entities,
created by an admin, objects are created on demand per file.

> ...and why can you not use a blocks based layout for said use case?

The blocks layout is even less capable then files-layout in it's toplogies
and it suffers from the same basic problem, volume based device map not
file-based.

And blocks layout has no fencing at all, who wants that headache?

Cheers
Boaz

---

From: "Haynes, Tom" <[email-masked]>
Date: Thu, 11 Oct 2012 00:14:53 +0000

> between client-to-MDS and client-to-DS.

...

> around saying the same arguments, That's it I'm done. Bye

Boaz,

How do you reconcile your statements that there is:

1) A secret protocol

2) old gear, no new software stacks

Either the MDS can communicate with the old gear or
it cannot.

You should read the draft with an open mind - I did. And then
follow this thread based on what is in the draft, not on
what you may know from earlier discussions.

Thanks,
Tom

PS: I will be sure to have dinner with you in a couple of weeks. We
can argue this out first and then who will pick up the tab! :->

---

From: Benny Halevy <[email-masked]>
Date: Thu, 11 Oct 2012 20:44:12 +0200

> switched union might be cleaner.

sure.

> to try other operations before establishing the session.

That applies to the files layout.
In our case EXCHGID4_FLAG_USE_NON_PNFS would be more appropriate.

> SHOULDN'T try to OPEN the file, which it ought never do on a

True.  Since there is no requirement for a back-end protocol the client
should open the file.

Benny