Subject: Re: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt

From: "Matt W. Benjamin" <[email-masked]>
Date: Wed, 10 Oct 2012 19:28:58 -0400 (EDT)

Hi,

It is clear to me at this point that the two broad usages that have been proposed are confusing when discussed together, whether or not they can have a common solution.

----- "Trond Myklebust" <[email-masked]> wrote:

> 
> I don't see that we would need to. It has been quite a few years since
> I
> saw a NAS unit that didn't also support at least one block protocol
> such
> as iSCSI (or even FCP). I'm certainly not aware of a NAS that
> supports
> NFSv4, but not iSCSI.

That appears to be special pleading for multi-protocol NAS products (and below).

> 
> > Where is the use case?
> 
> ...and why can you not use a blocks based layout for said use case?

Someone can, taking consideration of all the trade-offs that entails.  But frankly, I find that argument a little surprising.  Do you doubt the relevance of making NFSv4 itself a first-class access protocol for new storage systems such as Ceph, Lustre, glusterfs, etc, etc, to this working group?  Do you believe the feasibility of such a goal to be in doubt?  I do not.

> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> [email-masked]
> www.netapp.com
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 10 Oct 2012 23:35:01 +0000

> -----Original Message-----
> From: Matt W. Benjamin [mailto:[email-masked]]
> Sent: Wednesday, October 10, 2012 7:29 PM
> To: Myklebust, Trond
> Cc: nfsv4 list; Haynes, Tom
> Subject: Re: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt
> 
> Hi,
> 
> It is clear to me at this point that the two broad usages that have been
> proposed are confusing when discussed together, whether or not they can
> have a common solution.
> 
> > I saw a NAS unit that didn't also support at least one block protocol
> > such as iSCSI (or even FCP). I'm certainly not aware of a NAS that
> > supports NFSv4, but not iSCSI.
> 
> Someone can, taking consideration of all the trade-offs that entails.  But
> frankly, I find that argument a little surprising.  Do you doubt the relevance of
> making NFSv4 itself a first-class access protocol for new storage systems such
> as Ceph, Lustre, glusterfs, etc, etc, to this working group?  Do you believe the
> feasibility of such a goal to be in doubt?  I do not.

I have yet to see any evidence that any of the things Boaz is talking about is feasible. All I see at this point is a lot of talk about grand generalities about how people want to export all these fancy filesystems. Every time I try to request that people nail down what they need to do that, I get yet more generalities, and no specifics.

At this point, discussion of Benny's draft seems moot.

Trond

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 10 Oct 2012 23:48:38 +0000

> From: Myklebust, Trond
> Sent: Wednesday, October 10, 2012 7:35 PM
> To: 'Matt W. Benjamin'
> Subject: RE: [nfsv4] thoughts on draft-bhalevy-nfs-obj-00.txt
> 
> > It is clear to me at this point that the two broad usages that have
> > been proposed are confusing when discussed together, whether or not
> > they can have a common solution.
> >
> > > I don't see that we would need to. It has been quite a few years
> > > since I saw a NAS unit that didn't also support at least one block
> > > protocol such as iSCSI (or even FCP). I'm certainly not aware of a
> > > NAS that supports NFSv4, but not iSCSI.
> >
> > Someone can, taking consideration of all the trade-offs that entails.
> > But frankly, I find that argument a little surprising.  Do you doubt
> > the relevance of making NFSv4 itself a first-class access protocol for
> > new storage systems such as Ceph, Lustre, glusterfs, etc, etc, to this
> > working group?  Do you believe the feasibility of such a goal to be in doubt?
> I do not.
> 
> I have yet to see any evidence that any of the things Boaz is talking about is
> feasible. All I see at this point is a lot of talk about grand generalities about
> how people want to export all these fancy filesystems. Every time I try to
> request that people nail down what they need to do that, I get yet more
> generalities, and no specifics.

Let me correct that: I've seen what appears to be a beginning of a requirement from you. If you'd like to pursue defining requirements, and educating us on what you need in order to export CEPH (and other filesystems), then I'm more than happy to listen to that.

What I don't want to do, is pursue a discussion on the basis of "yea or nea on Benny's proposal", as I have no understanding of what his goals are, or why I should prefer his document to any other proposal.
 
> At this point, discussion of Benny's draft seems moot.

See above.

Trond

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 17:09:50 -0700

>> At this point, discussion of Benny's draft seems moot.

Thanks, this is exactly what I feel, moot. I speak but I guess
no voice is going out, proof that no one hears.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 11 Oct 2012 00:16:21 +0000

> From: Boaz Harrosh [mailto:[email-masked]]
> Sent: Wednesday, October 10, 2012 8:10 PM
> To: Myklebust, Trond
> Cc: Matt W. Benjamin; Haynes, Tom; nfsv4 list
> Thanks, this is exactly what I feel, moot.

Of no practical value or meaning?

   http://dictionary.reference.com/browse/Moot

>  I speak but I guess no voice is going out, proof that no one hears.

You're not answering the real question, which is "describe the problem in detail".

You're arguing for/against a specific solution instead of arguing the specifics of the problem.

Trond

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 17:46:14 -0700

> Of no practical value or meaning?

Well ye that too, but I was thinking mute, I guess

> You're not answering the real question, which is "describe the problem in detail".

I thought I did just that. "per file device table", that is the problem at hand.
Simple problem really ... And I did give examples

> You're arguing for/against a specific solution instead of arguing the specifics of the problem.

And that too. It was a long discussion. I explained the problem as well as specifics

> Trond

But I guess that's the end of that

Cheers
Boaz

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 17:54:01 -0700

> Simple problem really ... And I did give examples

Correction "per file topology"

By topology I mean:
 * striping/mirroring/raid type
 * stripe_unit size
 * .... (There is more see pnfs-objects)
 * Device table

And each file has it's own unique "Topology" the device-table is only one of
the things that can change per-file.

Thanks
Boaz

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 11 Oct 2012 01:00:13 +0000

> Sent: Wednesday, October 10, 2012 8:46 PM
> To: Myklebust, Trond

I saw the "per file device table" question. If that is the _only_ issue raised by the problem of wanting to export CEPH/Lustre/GlusterFS/... filesystems efficiently via pNFS, then let's proceed to discuss how to solve that. I'm more than happy to do so, and would definitely have a counter-proposal to Benny's draft.

If, however, there are other issues that need to be addressed, then let's get them laid out first.

Trond

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 10 Oct 2012 18:24:24 -0700

On 10/10/2012 06:00 PM, Myklebust, Trond wrote:
> I saw the "per file device table" question. If that is the _only_
> issue raised by the problem of wanting to export
> CEPH/Lustre/GlusterFS/... filesystems efficiently via pNFS, then
> let's proceed to discuss how to solve that. I'm more than happy to do
> so, and would definitely have a counter-proposal to Benny's draft.
> 

You answered the wrong mail, I had a correction on that. My correction said
"topology". All parameters not only "device table" can change per file.

* And also the need for mirroring/duplication raid and raid groups.

* And also the use of old filers

> If, however, there are other issues that need to be addressed, then let's get them laid out first.

Benny, did I miss anything?

Cheers
Boaz

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 11 Oct 2012 01:34:55 +0000

> Sent: Wednesday, October 10, 2012 9:24 PM
> To: Myklebust, Trond
> > CEPH/Lustre/GlusterFS/... filesystems efficiently via pNFS, then let's
> > proceed to discuss how to solve that. I'm more than happy to do so,
> > and would definitely have a counter-proposal to Benny's draft.
> >
> 

OK. I'm assuming that the OSD layout type requirements doc discusses this in sufficient detail that we can use that as our point of reference?

> * And also the use of old filers

This I have no documentation for, and is not anything that I recognise from the OSD, files or block layouts requirement documents.

Not touching this issue unless I get a proper problem description (and NO, your emails have done nothing to illuminate the problem)...

---

From: Benny Halevy <[email-masked]>
Date: Thu, 11 Oct 2012 21:17:26 +0200

> What I don't want to do, is pursue a discussion on the basis of "yea or nea on Benny's proposal", as I have no understanding of what his goals are, or why I should prefer his document to any other…

Sorry for the late response.

Tonian's main use case is clustering together standalone NAS systems
to achieve scale out of capacity and performance and applying flexible
management policies affecting per-file data location.

Therefore we require require the flexibility of the object layout striping/mirroring model
and a pragmatic approach to fence off unreachable clients and good enough security level
without requiring a mandatory MDS/DS back-end protocol.

I agree this should be documented.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 11 Oct 2012 21:22:03 +0200

> Benny, did I miss anything?

That's pretty much the gist of the problem :)

Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 11 Oct 2012 21:23:30 +0200

> Not touching this issue unless I get a proper problem description (and NO, your emails have done nothing to illuminate the problem)...

I hope to capture that more clearly.
Wondering about the IETF process, should that be part of the layout spec.
or be documented as a separate informational RFC in your opinion?

Benny

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 11 Oct 2012 19:42:59 +0000

> From: Benny Halevy [mailto:[email-masked]]
> Sent: Thursday, October 11, 2012 3:24 PM
> To: Myklebust, Trond
> Cc: Boaz Harrosh; Haynes, Tom; nfsv4 list
> >>> let's proceed to discuss how to solve that. I'm more than happy to
> >>> do so, and would definitely have a counter-proposal to Benny's draft.
> >>>
> >>
> >> You answered the wrong mail, I had a correction on that. My
> >> correction said "topology". All parameters not only "device table" can
> change per file.
> >>
> > OK. I'm assuming that the OSD layout type requirements doc discusses this
> in sufficient detail that we can use that as our point of reference?
> >
> > This I have no documentation for, and is not anything that I recognise from
> the OSD, files or block layouts requirement documents.
> >
> > Not touching this issue unless I get a proper problem description (and NO,
> your emails have done nothing to illuminate the problem)...
> >
> 

I'd be fine with it being a separate RFC, as long as there is something I can refer to when trying to understand the intent of the protocol. Documenting it in the layout spec itself is fine too...

The way we've tried to do this in the NFSv4.2 process is to have documentation+motivation in the personal drafts, and then when Tom has pulled stuff into the NFSv4.2 draft spec, he has usually dropped the motivation in order to end up with a shorter document for publishing.

Trond

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 11 Oct 2012 20:11:57 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Benny Halevy
> Sent: Thursday, October 11, 2012 12:24 PM
> To: Myklebust, Trond
> Cc: Haynes, Tom; nfsv4 list

[Spencer] 

As mentioned in my other mail, the I-D needs to include the motivation. 

How we deal with that content moving forward is up to the final "packaging" for the work and it may be included in any eventual RFC that might be publish.

Better yet would to have some of that content presented directly here in the maillist so that we can generate feedback and avoid the delay of inclusion in an I-D (editing/publication delay).

Spencer