Subject: Re: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 9 Oct 2012 23:49:19 -0400 (EDT)

Hi Trond,

Yes, I agree, we need to step back and try to better articulate requirements.  We'll make an effort to do that, from our end (after more sleep).

I think we all hold the premise that it is in principle good (good for the protocol, good engineering) to explore improvements/enhancements that align well with existing design decisions and reuse existing protocol and implementation, when possible, building on successes.

So to be fair, I am myself trying to avoid special pleading.  On the other hand, I'd like not to run into a dead end.  Our implementation experience we think confirms that we -require- more striping flexibility than files provides.  I feel the ability to cover the Ceph design space is an indicator that is important.  Probably much more convincing in general is the design and implementation experience at Panasas and elsewhere, with the objects layout (since it is more extensive).  Meanwhile, since we plan to still be deploying -NFSv4- when we are finished, we feel that -requires- ability to use the rest of the protocol (RPCSEC_GSS, the layout concept, etc) that is already developed.  Brent's summary captures what Adam and I have been trying to say, I think, extremely well.

Regards,

Matt

----- "Trond Myklebust" <[email-masked]> wrote:

> Right, but there appears to be no consensus yet on what the
> requirements are. Boaz seems to be speaking one language, you are
> speaking a different one, and Benny is (as of yet) only discussing
> protocols.
> While you have stated that you want to export CEPH, I have yet to see
> a discussion of what minimum set of functionality that _requires_. I
> have also yet to see a motivation of why a protocol extension to allow
> exporting of a CEPH filesystem in particular is of interest to us
> all.
> 
> So could we please start a _requirements_ discussion before discussing
> implementation detail merits!
> 
> > -----Original Message-----
> > From: Matt W. Benjamin [mailto:[email-masked]]
> > Sent: Tuesday, October 09, 2012 10:43 PM
> > To: Myklebust, Trond
> > Cc: Haynes, Tom; nfsv4 list; Boaz Harrosh
> > Subject: Re: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt
> > 
> > Hi Trond,
> > 
> > As I have stated, the memory hogging claim is specious.  As well,
> there is and
> > no need (or motivation) to invoke CRUSH.  While we are obviously
> working in
> > this area specifically, hard-wiring a design that narrowly would
> constitute a
> > regression for us.  Meanwhile, the pNFS model is already more
> general and
> > well articulated.  We are discussing refinements to that model,
> which have
> > not gone unmotivated in this discussion.
> > 
> > Matt
> > 
> > ----- "Trond Myklebust" <[email-masked]> wrote:
> > 
> > > On Tue, 2012-10-09 at 18:46 -0700, Boaz Harrosh wrote:
> > > > On 10/09/2012 11:15 AM, Myklebust, Trond wrote:
> > > > >
> > > > > The main cost on the wire would be more DEVICEINFO calls,
> however
> > > I'm
> > > > > not sure how Matt can avoid either a high DEVICEINFO or a
> high
> > > LAYOUTGET
> > > > > balance sheet unless he wants to write a layout type that
> > > directly
> > > > > exports the ceph CRUSH algorithm.
> > > > >
> > > >
> > > > You completely missed the point. It is to do with state. Not
> memory
> > > > or bandwidth at all.
> > >
> > > I'm missing several points, and it is mainly to do with what
> problems
> > > people are trying to solve. As I said earlier, I don't know what
> the
> > > problem statement is...
> > >
> > > > a DEVICEINFO is not associated with any file, it is a global
> list
> > > > that today doesn't ever shrink. Layouts are per file, and if
> the
> > >
> > > CB_NOTIFY_DEVICEID...
> > >
> > > > striping is done per file (or per object) then that is a file's
> > > > resource. RETURN-ON-CLOSE will discard all these layouts. and
> recall
> > > > will trash all these, and commit, and so on per-file.
> > > > In your case the server resources as well as the clients will
> > > > explode, because you have a global list where you should have a
> per
> > > > file resource management. Also note all the fine grained layout
> > > > state protocol, helping you sync with clients, none of that is
> done
> > > > on deviceids. A per-file striping will need all the layout
> protocol
> > > > goodies to survive. Anything less is an hack for the lab only.
> > > >
> > > > It's simple. Is your cluster a per-export striping, global to
> all
> > > > files then yes, device_info. If your cluster is per-file
> striping
> > > > then it is more natural in the layout which is per-file.
> > >
> > > Per-file striping is going to be a memory hog unless I can
> compress it
> > > into something like CRUSH. Why would I prefer the memory hog?
> > >
> > > IOW: what is the goal here? To be able to export arbitrary
> striping
> > > algorithms, or to enable limited but functionally sufficient
> > > alternatives (for specific workloads?)? What data do you have to
> > > suggest that we need either one (or both)?
> > >
> > > The total set of pNFS layout protocols is infinite, as far as I
> can
> > > tell. Somewhere, someone has to be able to tell me which subset
> of
> > > these protocols are solving interesting problems...
> > >
> > > --
> > > Trond Myklebust
> > > Linux NFS client maintainer
> > >
> > > NetApp
> > > [email-masked]
> > > www.netapp.com
> > > _______________________________________________
> > > nfsv4 mailing list
> > > [email-masked]
> > > https://www.ietf.org/mailman/listinfo/nfsv4
> > 
> > --
> > Matt Benjamin
> > The Linux Box
> > 206 South Fifth Ave. Suite 150
> > Ann Arbor, MI  48104
> > 
> > http://linuxbox.com
> > 
> > tel. 734-761-4689
> > fax. 734-769-8938
> > cel. 734-216-5309

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Wed, 10 Oct 2012 23:43:12 -0400 (EDT)

Hi,

----- "Trond Myklebust" <[email-masked]> wrote:

> > * And also the use of old filers
> 
> This I have no documentation for, and is not anything that I recognise
> from the OSD, files or block layouts requirement documents.
> 
> Not touching this issue unless I get a proper problem description (and
> NO, your emails have done nothing to illuminate the problem)...

This really is a very different sort of issue from the striping model, and I feel a bit bad about having hijacked the discussion of these ideas from the v2 draft, which Benny I'm certain will expand on to address motivation.

> 
> > > If, however, there are other issues that need to be addressed,
> then let's
> > get them laid out first.
> > >
> > 
> > Benny, did I miss anything?

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309