Subject: Re: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 9 Oct 2012 12:32:22 -0400 (EDT)

Hi Trond,

The only way in which the current pNFS files layout fails -for us- (Linux Box, ) is in its specification of the striping pattern to be in the device, rather than in the layout, as is done in objects.

As we presented at IETF 84, the files approach is -very- badly suited to Ceph and, we believe, a whole class of storage systems which have many striping patterns (in Ceph, a unique striping pattern per file, leading to an explosion of pNFS devices to represent a single object).

Being fair, our considerations are quite distinct from Benny's, and perhaps our natural home remains the files layout, since the above issue is the issue with it I believe we have.  If we could find a way to get the striping flexibility we require in a files layout, it might be easier to discuss the motivations for an extended objects layout separately.

Matt

----- "Trond Myklebust" <[email-masked]> wrote:

> On Tue, 2012-10-09 at 11:30 -0400, Adam C. Emerson wrote:
> 
> > Also, in the case of a server that isn't aware that it's
> participating
> > in pNFS (the use case you describe in 13.5 where the MDS changes
> > permissions on the server) it's not clear to me that the client
> > SHOULDN'T try to OPEN the file, which it ought never do on a
> > cooperating pNFS DS.
> 
> Alright, so what does all this extra complexity and extra
> client-server
> back and forth interaction buy us?
> 
> Where's the killer app that cannot be satisfied with today's pNFS
> technologies?
> 
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> [email-masked]
> www.netapp.com
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 9 Oct 2012 18:05:49 +0000

> As we presented at IETF 84, the files approach is -very- badly suited to Ceph and, we believe, a whole class of storage systems which have many striping patterns (in Ceph, a unique striping pattern…

Is the objection that you need both a data structure for the layout and the devices?

While in practice we see pretty fixed layout/device mappings for current pNFS file
servers, I know we had been considering each file having a different device
mapping with just what was in 5661:

http://www.nfsv4bat.org/Documents/ConnectAThon/2008/spe.pdf

I never did a memory footprint analysis to see what the cost was to have
each file have a unique device id mapping per file.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 9 Oct 2012 18:15:52 +0000

> each file have a unique device id mapping per file.

I don't think that the memory cost would be so huge on the client.
Keeping the information in the device id or the layout structure would
end up being more or less the same to us. The only real difference is a
slight penalty incurred when looking up the deviceids...

The main cost on the wire would be more DEVICEINFO calls, however I'm
not sure how Matt can avoid either a high DEVICEINFO or a high LAYOUTGET
balance sheet unless he wants to write a layout type that directly
exports the ceph CRUSH algorithm.

NetApp

---

From: "Adam C. Emerson" <[email-masked]>
Date: Tue, 09 Oct 2012 14:20:44 -0400

Mr. Haynes,

At Tue, 9 Oct 2012 18:05:49 +0000,
> each file have a unique device id mapping per file.

Our objection is that under this system we can potentially need a
unique device for every file.  We would prefer a system where, for
example, the device for the files layout type contained just a list of
data servers, and the stripe indices were in the layout.  We also
believe we could make productive use of the richer capabilities
something like pnfs_obj_data_map4 provides over asimple stripe index
array.

Respectfully yours,

Adam C. Emerson                                  <[email-masked]>

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 9 Oct 2012 14:29:01 -0400 (EDT)

Hi Trond, Tom,

I think the issue isn't reducible to memory costs.  It's just inefficient not to share devices across (many) files.

We would be willing to write a layout that specifically dealt with this issue.  We'd strongly hope to re-use specification from files for this layout specifically (as opposed to one we might write that knew our equivalent to CRUSH), since the issues we're discussion here really do seem to be general, not actually specific to Ceph.

Matt

----- "Trond Myklebust" <[email-masked]> wrote:

> > > The only way in which the current pNFS files layout fails -for us-
> (Linux Box, ) is in its specification of the striping pattern to be in
> the device, rather than in the layout, as is done in objects.
> > > 
> > > As we presented at IETF 84, the files approach is -very- badly
> suited to Ceph and, we believe, a whole class of storage systems which
> have many striping patterns (in Ceph, a unique striping pattern per
> file, leading to an explosion of pNFS devices to represent a single
> object).
> > 
> > 

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 9 Oct 2012 18:39:37 +0000

> something like pnfs_obj_data_map4 provides over asimple stripe index

So if the object layout striping mechanism works for you, what is
stopping you from putting an OSD frontend to ceph?

NetApp

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 9 Oct 2012 14:43:16 -0400 (EDT)

Hi Trond,

Several reasons, including that 1) Ceph doesn't use the T10 protocol natively;  2) we wish to take advantage of GSS-based RPC security; 3) we want in general the full semantics of the files layout.

Regards,

Matt

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 9 Oct 2012 19:04:10 +0000

> Several reasons, including that 1) Ceph doesn't use the T10 protocol natively;  2) we wish to take advantage of GSS-based RPC security; 3) we want in general the full semantics of the files layout.

So while Benny's new draft solves 1) for you, it does not solve 2) and 3) ?

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 9 Oct 2012 15:16:24 -0400 (EDT)

Hi Tom,

Effectively, it seems as if Benny's draft solves 1) relative to the objects draft, and with some adjustments, seems not too far from solving problem 2) (but doesn't as written).   But as I see more clearly after discussion, it essentially creates problem 3.  Moreover, as we discussed this more over here, Adam was able to make a rather strong case that drawing all the semantics of the files layout into the objects v2 draft wrenches it a great deal, and might sit uneasily with the files layout in implementation, while a files v2 layout with some striping adjustments might be a lot more natural.

Matt

----- "Tom Haynes" <[email-masked]> wrote:

> > Several reasons, including that 1) Ceph doesn't use the T10 protocol
> natively;  2) we wish to take advantage of GSS-based RPC security; 3)
> we want in general the full semantics of the files layout.
> > 
> > Regards,
> > 
> > Matt
> 

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Welch, Brent" <[email-masked]>
Date: Tue, 9 Oct 2012 22:58:01 +0000

IMHO, and this isn't really what the draft is getting to (all the way), is that the objects layout
has interesting and compelling layout options that are not present in the files layout.  We tried early
to make the striping stuff generic, and sit above the layout drivers.  But, that was too radical.  So,
I'm not surprised to see folks wanting different storage access protocols (e.g., NFSv3 or NFSv4*) composed
with more sophisticated striping algorithms.  In the long run, I think it would be better to have that
kind of orthogonality - especially in the implementations, if not also the standards.  This draft is
a step in the right direction.

--
Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Matt W. Benjamin
Sent: Tuesday, October 09, 2012 12:16 PM
To: Tom Haynes
Cc: Trond Myklebust; nfsv4 list
Subject: Re: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt

Hi Tom,

Matt

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 9 Oct 2012 23:55:26 +0000

> From: Welch, Brent [mailto:[email-masked]]
> Sent: Tuesday, October 09, 2012 6:58 PM
> To: Matt W. Benjamin; Haynes, Tom
> Cc: Myklebust, Trond; nfsv4 list
> Subject: RE: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt
> 
> IMHO, and this isn't really what the draft is getting to (all the way), is that the
> objects layout has interesting and compelling layout options that are not
> present in the files layout.  We tried early to make the striping stuff generic,
> and sit above the layout drivers.  But, that was too radical.  So, I'm not
> surprised to see folks wanting different storage access protocols (e.g., NFSv3
> or NFSv4*) composed with more sophisticated striping algorithms.  In the
> long run, I think it would be better to have that kind of orthogonality -
> especially in the implementations, if not also the standards.  This draft is a
> step in the right direction.

I have no metric whatsoever for understanding whether or not that might be the case.

Benny describes a potential protocol implementation without a single sentence anywhere explaining why I might be even remotely interested. It jumps straight from '1. Introduction' (which tells me what a pNFS layout and an "object" is) to '2. XDR description of the Objects-Based Layout Protocol'.

If the object here is to add a more generic striping, then let's get a description of what problems people want this to solve, and why.

IOW, before we laud the merits of the protocol implementation:

 1) What would a problem statement rfc for a "pNFS files layout v2" look like?
 2) What compelling reasons would implementers of the existing files layout have for implementing the new layout type?

Trond

> Effectively, it seems as if Benny's draft solves 1) relative to the objects draft,
> and with some adjustments, seems not too far from solving problem 2) (but
> doesn't as written).   But as I see more clearly after discussion, it essentially
> creates problem 3.  Moreover, as we discussed this more over here, Adam
> was able to make a rather strong case that drawing all the semantics of the
> files layout into the objects v2 draft wrenches it a great deal, and might sit
> uneasily with the files layout in implementation, while a files v2 layout with
> some striping adjustments might be a lot more natural.
> 
> Matt
>

---

From: Boaz Harrosh <[email-masked]>
Date: Tue, 9 Oct 2012 17:11:14 -0700

On 10/09/2012 11:43 AM, Matt W. Benjamin wrote:
> Hi Trond,
> 
> Several reasons, including that 
> 1) Ceph doesn't use the T10 protocol natively;
> 2) we wish to take advantage of GSS-based RPC security;
> 3) we want in general the full semantics of the files layout.
> 

Sorry for being slow. But what exactly do you mean by 3?
What "full semantics of the files layout" are you missing
and that cannot be easily implemented in Benny's draft?

For example files-layout stateids. Well that's a missing
fixture in Benny's draft that I think you where supposed
to fix. Bennie's draft Has 3 variants of NFS access to
DSs. NFSv3 NFSv4 and NFSv4.1-MDS there is a missing 4th
one that we want which is NFSv4.1-DS.

With the NFSv4.1-DS variant the MDS will pass a stateid
as part of the object's caps buffer. Also in this mode
the client will take the special "DS" role and for example
will not send GETATTR op with every WRITE compound, and so
on ...
Also in this mode with set of a flag, say gid/uid == -1
the client will present it's file credentials instead of
the passed in creds from MDS.

Cheers
Boaz

---

From: Boaz Harrosh <[email-masked]>
Date: Tue, 9 Oct 2012 18:46:50 -0700

> exports the ceph CRUSH algorithm.

You completely missed the point. It is to do with state. Not memory
or bandwidth at all.

a DEVICEINFO is not associated with any file, it is a global list
that today doesn't ever shrink. Layouts are per file, and if the
striping is done per file (or per object) then that is a file's
resource. RETURN-ON-CLOSE will discard all these layouts. and
recall will trash all these, and commit, and so on per-file.
In your case the server resources as well as the clients will
explode, because you have a global list where you should have
a per file resource management. Also note all the fine grained
layout state protocol, helping you sync with clients, none of that
is done on deviceids. A per-file striping will need all the layout
protocol goodies to survive. Anything less is an hack for the lab
only.

It's simple. Is your cluster a per-export striping, global to all
files then yes, device_info. If your cluster is per-file striping
then it is more natural in the layout which is per-file.

Cheers
Boaz

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 10 Oct 2012 02:22:25 +0000

> You completely missed the point. It is to do with state. Not memory

I'm missing several points, and it is mainly to do with what problems
people are trying to solve. As I said earlier, I don't know what the
problem statement is...

> that today doesn't ever shrink. Layouts are per file, and if the

CB_NOTIFY_DEVICEID...

> then it is more natural in the layout which is per-file.

Per-file striping is going to be a memory hog unless I can compress it
into something like CRUSH. Why would I prefer the memory hog?

IOW: what is the goal here? To be able to export arbitrary striping
algorithms, or to enable limited but functionally sufficient
alternatives (for specific workloads?)? What data do you have to suggest
that we need either one (or both)?

The total set of pNFS layout protocols is infinite, as far as I can
tell. Somewhere, someone has to be able to tell me which subset of these
protocols are solving interesting problems...

NetApp

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 9 Oct 2012 22:43:29 -0400 (EDT)

Hi Trond,

As I have stated, the memory hogging claim is specious.  As well, there is and no need (or motivation) to invoke CRUSH.  While we are obviously working in this area specifically, hard-wiring a design that narrowly would constitute a regression for us.  Meanwhile, the pNFS model is already more general and well articulated.  We are discussing refinements to that model, which have not gone unmotivated in this discussion.

Matt

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 10 Oct 2012 02:56:24 +0000

Right, but there appears to be no consensus yet on what the requirements are. Boaz seems to be speaking one language, you are speaking a different one, and Benny is (as of yet) only discussing protocols.
While you have stated that you want to export CEPH, I have yet to see a discussion of what minimum set of functionality that _requires_. I have also yet to see a motivation of why a protocol extension to allow exporting of a CEPH filesystem in particular is of interest to us all.

So could we please start a _requirements_ discussion before discussing implementation detail merits!

> From: Matt W. Benjamin [mailto:[email-masked]]
> Sent: Tuesday, October 09, 2012 10:43 PM
> To: Myklebust, Trond
> Cc: Haynes, Tom; nfsv4 list; Boaz Harrosh
> As I have stated, the memory hogging claim is specious.  As well, there is and
> no need (or motivation) to invoke CRUSH.  While we are obviously working in
> this area specifically, hard-wiring a design that narrowly would constitute a
> regression for us.  Meanwhile, the pNFS model is already more general and
> well articulated.  We are discussing refinements to that model, which have
> not gone unmotivated in this discussion.
> 
> Matt
> 
> > > resource. RETURN-ON-CLOSE will discard all these layouts. and recall
> > > will trash all these, and commit, and so on per-file.
> > > explode, because you have a global list where you should have a per
> > > file resource management. Also note all the fine grained layout
> > > state protocol, helping you sync with clients, none of that is done
> > > on deviceids. A per-file striping will need all the layout protocol
> > > goodies to survive. Anything less is an hack for the lab only.
> > >
> > alternatives (for specific workloads?)? What data do you have to
> > suggest that we need either one (or both)?
> >
> > tell. Somewhere, someone has to be able to tell me which subset of
> > these protocols are solving interesting problems...
> >
> > --
> > Trond Myklebust

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 11:39:53 -0700

> people are trying to solve. As I said earlier, I don't know what the

The problem is that many cluster topologies are not describable
by files-layout, but are easily describable by pnfs-objects
layout. Including mirroring, duplication, raid4/5/6 ...

But current pnfs-objects layout mandates OSD T10 protocol. Which
requires two protocol stacks implemented in a cluster. An NFS server
for MDS and a T10-OSD server for DSs. If on the other hand, OSDs could
be converted to speak NFS, then all servers in the system MDS+DSs
deploy the same server protocol with a single software component.
(And can still enjoy the cluster topologies they deploy now)

The other problem this tries to solve is that old NFS filers in
the field today that have been bought with hard cash and still perform
very well, must never participate in a pNFS cluster. The motivation
of this draft is to also include all these filers in a very capable
very scaleable cluster.

>> that today doesn't ever shrink. Layouts are per file, and if the

Not enough, and not scalable, and not safe. 

CB_NOTIFY_DEVICEID is not governed by a state_id, seqid so there might
be races between the returns and the recalls. also we have all these
implicit returns of ROC, no_maching_layout, and so on ...

<snip>
> into something like CRUSH. Why would I prefer the memory hog?

Well than pnfs-objects is already a memory hog? (Rhetorical)

With object layout I'm able to saturate a 10G link with a single dd
and am able to saturate a 100G capable cluster with 10 clients, and
performance measurements for both large files and small files show
great promise. So I do not see that hog of yours. Actually it is
not at all that big, and it is always proportional to the file size
so it disappears in the noise.

> that we need either one (or both)?

Both! (see above) All of CEPH, Luster, PVFS, Gluster, GPFS ... can be
easily describable by objects-layout and cannot by files-layout.

And lots and lots of new ones could be easily supported by objects-layout
without changing a single bit of code or a single char at STD. So the
goal is to deploy the more general objects-layout math that is a general
decoder for any imaginable layout to date, in a single specification.

> tell. 

And objects-layout math can describe infinite number of those, I'm
yet to find a layout not supported by the objects-layout math.

> Somewhere, someone has to be able to tell me which subset of these

Exactly, What Benny's draft is trying to do is to *not* add another
wire protocol. T10-OSD is already there, and NFSv3, NFSv4, NFSv4.1
which are not new, also as part of the objects-layout math.
(Unlike some other drafts floating around that propose a new wire
 protocol, and a new client stack)

Cheers
Boaz

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 10 Oct 2012 21:11:05 +0000

> decoder for any imaginable layout to date, in a single specification.

It isn't doing that. It is adding a 'NFS' mode to the objects layout. So
we end up with 2 forms of pNFS file based layouts, one of which is for
some reason a versioned subtype of the object layout type. Why?
There are all sorts of discussions that are being short-circuited by
camouflaging this as an object layout, and pretending that it is somehow
covered by the original OSD layout requirements.

For instance, the original OSD layout requirements were very specific
about the use of T-10 capabilities as being the basis for the object
layout security model. Benny's draft removes the T-10 capabilities and
replaces them with SSVs. Now you are furthermore talking about exporting
filesystems that don't use T-10 capabilities as part of their security
model.
So where is the discussion of the consequences? Those certainly aren't
covered by the brief discussions in Sections 13.5.

Then there is the brief discussion of fencing. Again, no requirements,
but a suggestion that changing file ownership can do the job.
Really? So do I deduce from that that being able to access this
underlying NFS-exported filesystem natively or from a different client
is not part of the unspecified requirements list?
Oh, and for the record: ACLs aren't part of the NFSv2 or NFSv3
protocols, so the only valid value for 'ona_version' would appear to be
'4'.

>  protocol, and a new client stack)

This is clearly a new layout type, and needs to be treated as such. It
proposes new security and fencing models, apparently in order to
introduce new applications.

The draft needs to discuss all this in detail.

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Thu, 11 Oct 2012 21:05:33 +0200

On 2012-10-10 02:11, Boaz Harrosh wrote:
> as part of the object's caps buffer.

No need to pass a stateid in this mode.  The files layout model
assumes the client OPENs the file on the MDS and uses the open
(or a derived lock/deleg stateid) for IO operations sent to the
DS.

Benny

> Also in this mode

---

From: Boaz Harrosh <[email-masked]>
Date: Thu, 11 Oct 2012 14:02:04 -0700

> (or a derived lock/deleg stateid) for IO operations sent to the

No need, in NFSv4.1 Stand alone mode.

But we would like a 4th mode added, where the DS acts like an pNFS-files DS
with a stateid and no OPEN, with the EXCHGID_FLAG_USE_PNFS_DS.
(AND with no get_attrs in the WRITE compound)

Because in Ganesha codebase for example the DS_WRITE is a special
fast lane codepath that bypasses lots of checks and goes directly
to the filesystem.

We would like to enable a mode where when the MDS+DSs are all under
the same software version and have a way to communicate even just through
filehandle/stateid and bypass any kind of OPENs and state management
against DSs (Just like in files pNFS)

Imagine that a DS filehandle received in the layout is like an OSD
caps with a cartographic signature and caps structure. Hell it can
even have a caps version for fence off.  Because remember that DS
filehandles received on a layout are not NFS filehandles and need
not be unique or persistent.

> Benny

Thanks
Boaz