Subject: Re: [nfsv4] NFSv4.2 vs pNFS

From: Christoph Hellwig <[email-masked]>
Date: Mon, 3 Feb 2014 20:22:42 +0100

On Wed, Dec 04, 2013 at 02:39:07PM +0100, Christoph Hellwig wrote:
> This seems sensible, but is not obvious from the naming of the file layout
> type.  I think it would be useful the 4.2 standard would include language
> to that extent in addition to the clarifications for the specific
> commands.  I could envision something like:
> 
> 
> X.Y.   NFSv4.2 as a Storage Protocol in pNFS: the File Layout Type
> 
> When using NFSv4.2 to speak to a pNFS MDS, the DSs are expected to use
> NFSv4.2 semantics for the their operation if those different from NFSv4.1
> 
> X.Y.1.  Operations Sent to NFSv4.2 Data Servers
> 
> In addition to the commands listed in [RFC5661], NFSv4.2 data servers may
> accept COMPOUND containing the following additional operations:
> READ_PLUS, WRITE_PLUS and SEEK, which will be treated like the subset
> specified as "data-server I/O operations" in [RFC5661].
> 
> Additional details on the implementation of these operations in a pNFS
> context are documented in the operation specific sections below.

Does anyone disagree with the need for these sections or should I prepare
a patch to add them?

---

From: Jim Zelenka <[email-masked]>
Date: Mon, 3 Feb 2014 14:39:36 -0500

This implies that a 4.2 MDS cannot use a 4.1 or 4.0 DS to store data. Is 
that intended? It seems desirable to be able to provide a MDS that can 
use down-revision (or even third-party) DSs.

-Jim Zelenka
Chief Software Architect
Panasas, Inc
www.panasas.com

On 2/3/2014 2:22 PM, Christoph Hellwig wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Adamson, Andy" <[email-masked]>
Date: Tue, 4 Feb 2014 15:21:49 +0000

On Feb 3, 2014, at 2:39 PM, Jim Zelenka <[email-masked]> wrote:

> This implies that a 4.2 MDS cannot use a 4.1 or 4.0 DS to store data.

I do not read it that way. “When using NFSv4.2 to speak to a pNFS MDS" != 4.2 MDS cannot use NFSv4.1/0. Anyway, we can make it very clear that the new 4.2 language does not limit the use of earlier NFSv4 versions.

> Is that intended? It seems desirable to be able to provide a MDS that can use down-revision (or even third-party) DSs.
> 
> -Jim Zelenka
> Chief Software Architect
> Panasas, Inc
> www.panasas.com
> 
> 

I like it.

—>Andy

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 4 Feb 2014 16:33:08 +0100

> I do not read it that way. “When using NFSv4.2 to speak to a pNFS MDS" != 4.2 MDS cannot use NFSv4.1/0. Anyway, we can make it very clear that the new 4.2 language does not limit the use of earlier…

I guess the language wasn't clear enough, but I also have a hard time coming up
with good language for it.  Maybe the best is to iterate the cases:

A file layout handed out from a NFSv4.2 server may refer to a DS that either
speaks NFSv4.1 as specified in RFC5661, or to a DS that implements additions
from NFSv4.2, in which case the rules in section X.Y.1. apply.

I don't think speaking NFSv4.0 is covered by the language in either RFC5661 or
NFSv4.2 and I can't really see how it's supposed to work.

Given that MDSes and DSes need an implementation specific sideband protocol
the idea of a DS from a different vendor doesn't seem feasible either.

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 4 Feb 2014 16:42:55 +0000

> the idea of a DS from a different vendor doesn't seem feasible either.

I think Benny would argue on that one with you. :-)

The language in RFC 5661 could be explained a bit better - i.e., where does it
describe the pNFS requirements for layouts versus those for file layouts?

You could meet the requirements for layouts and deliver a frankenstein.

But it is the things which are not defined in the spec and customers are now
used to which will kill you:

unified backup
snapshots
file deletion

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 4 Feb 2014 17:53:47 +0100

> describe the pNFS requirements for layouts versus those for file layouts?

Section 13.1. explicitly states in the section description that we are
using NFSv4.1 as the protocol to speak to the DS.  Not NFS, not NFS4 but
specificly NFSv4.1.  I think that's as explicit as it gets for this
specific layout.

> You could meet the requirements for layouts and deliver a frankenstein.

Of course you could create another layout that uses some form of NFS
as communication protocol to a DS.  That's not what I'm talking about here.
Unfortunately the inclusion of the NFSv4.1 file layout in RFC5661, and the
name that specificly says 4.1 makes all this a bit confusing.

And the whole intent of my addition to the 4.2 spec was to allow for using
4.2 features to a DS when using the LAYOUT4_NFSV4_1_FILES layout type.

---

From: Tom Talpey <[email-masked]>
Date: Tue, 4 Feb 2014 17:41:46 +0000

> -----Original Message-----
> From: nfsv4 [mailto:[email-masked]] On Behalf Of Christoph Hellwig
> Sent: Tuesday, February 4, 2014 11:54 AM
> To: Haynes, Tom
> Cc: Adamson, Andy; [email-masked]
> Subject: Re: [nfsv4] NFSv4.2 vs pNFS
> 
> > The language in RFC 5661 could be explained a bit better - i.e., where
> > does it describe the pNFS requirements for layouts versus those for file
> layouts?
> 
> Section 13.1. explicitly states in the section description that we are using
> NFSv4.1 as the protocol to speak to the DS.  Not NFS, not NFS4 but specificly
> NFSv4.1.  I think that's as explicit as it gets for this specific layout.

Section 13.1.1 goes on to make the specific normative requirement though, which is for "sessions". This is made without specific justification, but it's apparently for reliability and recovery.

There is no mention of what *other* NFSv4.1 attributes might be required, especially, any of the *optional* attributes. At a minimum, normative requirements are required here - beyond a simple "4.1" or "4.2" statement.
 
> Of course you could create another layout that uses some form of NFS as
> communication protocol to a DS.  That's not what I'm talking about here.

More importantly, it's critical to avoid specifying a non-interoperable Frankenstein. 4.2 features need to be explicitly required, or made explicitly optional, or whatever.

In other words, is the statement below "MAY accept", and if so, how does the client know? These need to be stated.

  > In addition to the commands listed in [RFC5661], NFSv4.2 data servers 
  > may accept COMPOUND containing the following additional operations:

Tom.

---

From: Jim Zelenka <[email-masked]>
Date: Tue, 4 Feb 2014 12:50:09 -0500

On 2/4/2014 11:53 AM, Christoph Hellwig wrote:
>
>   requirements for layouts versus those for file layouts?

You are correct. I shouldn't have mentioned using anything pre-4.1.

> 4.2 features to a DS when using the LAYOUT4_NFSV4_1_FILES layout type.

Right - I'm just wondering if the intent is to require that the DS 
support 4.2 if the MDS is 4.2. The general feedback appears to be that 
it should be okay to have a 4.1 DS, but that could also be me reading 
what I want to read.

I'm not attempting to make a business case for doing so. :-) My 
preference is to allow doing so, and I do feel that it would help if the 
language in the spec were more explicit either way.

-Jim Zelenka
Chief Software Architect
Panasas, Inc
www.panasas.com

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 4 Feb 2014 19:04:05 +0100

> In other words, is the statement below "MAY accept", and if so, how does the client know? These need to be stated.

All the operations listed below are optional in 4.2 and thus not required
to be implemented on a MDS or stand-alone server either, that's why I used the
MAY.  Any better way to word this?

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 4 Feb 2014 19:05:27 +0100

On Tue, Feb 04, 2014 at 12:50:09PM -0500, Jim Zelenka wrote:
> Right - I'm just wondering if the intent is to require that the DS support 
> 4.2 if the MDS is 4.2. The general feedback appears to be that it should be 
> okay to have a 4.1 DS, but that could also be me reading what I want to 
> read.
>
> I'm not attempting to make a business case for doing so. :-) My preference 
> is to allow doing so, and I do feel that it would help if the language in 
> the spec were more explicit either way.

To me supporting different protocol versions in a single cluster is asking
for trouble.  For 4.2 the only differences are newly added optional commands,
but I'm talking about the general problem.

That being said my major motivation here was to explicitly write down what
is allowed so that clients can rely on it.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 4 Feb 2014 13:15:27 -0500

> Section 13.1.1 goes on to make the specific normative requirement though, which is for "sessions". This is made without specific justification, but it's apparently for reliability and recovery.

That’s what I seem to remember from the IETF meetings. The other reasons I seem to recall for wanting NFSv4.1 only, were:

1) Requirement that DSes support security negotiation using SECINFO_NO_NAME, which is an NFSv4.1 operation only.
2) Nobody had the stomach to try to define pNFS DS semantics for NFSv4.0 or NFSv3.

> There is no mention of what *other* NFSv4.1 attributes might be required, especially, any of the *optional* attributes. At a minimum, normative requirements are required here - beyond a simple "4.1…

There is quite a bit of normative text in RFC5661, section 13.6 describing what operations a DS may, and may not, support. For NFSv4.2, we really only need to add the WRITE_PLUS, READ_PLUS, and SEEK operations.

I’d really like to avoid adding VERIFY/NVERIFY to the list of DS operations, though (see section 8.4 in NFSv4.2 draft 21). Why can’t we just require that the MDS must recall (or revoke) all outstanding layouts if/when the security label changes on the file?

> In other words, is the statement below "MAY accept", and if so, how does the client know? These need to be stated.

The client tries the operation, and if the server returns NFS4ERR_NOTSUPP, it tries the fallback READ or WRITE. Why would we need a separate negotiation method for DSes?

>> specified as "data-server I/O operations" in [RFC5661].

Cheers,
  Trond

---

From: Tom Talpey <[email-masked]>
Date: Tue, 4 Feb 2014 18:18:10 +0000

> From: Christoph Hellwig [mailto:[email-masked]]
> Sent: Tuesday, February 4, 2014 1:04 PM
> To: Tom Talpey
> Cc: Christoph Hellwig; [email-masked]
> > More importantly, it's critical to avoid specifying a non-interoperable
> Frankenstein. 4.2 features need to be explicitly required, or made explicitly
> optional, or whatever.
> >
> > In other words, is the statement below "MAY accept", and if so, how does
> the client know? These need to be stated.
> 
> All the operations listed below are optional in 4.2 and thus not required to be
> implemented on a MDS or stand-alone server either, that's why I used the

It needs more than rewording - it needs actual requirements. "MAY" is a rather weak term, and in the sentence above, it only applies to the server. How does the client detect this optional behavior by its peer? How is it to modify its own behavior in response? What is the protocol's justification for making it optional versus required? As written, the statement gives no guidance and therefore will not lead to interoperability. 

Tom.

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 5 Feb 2014 17:41:15 +0100

> It needs more than rewording - it needs actual requirements. "MAY" is a rather weak term, and in the sentence above, it only applies to the server. How does the client detect this optional behavior…

As Trond already mentioned we already clearly define how clients deal with
these optional commands in the main NFSv4.2 spec.  The question is just
how to best word the reference to it.  I agree that the MAY is not very
specific, and I'm open to rewording it.  In fact I suspect thay the point
has come where I made the intent clear and should leave the wording to
professional spec wranglers..

---

From: Tom Talpey <[email-masked]>
Date: Wed, 5 Feb 2014 20:23:02 +0000

> Sent: Wednesday, February 5, 2014 11:41 AM
> To: Tom Talpey
> Cc: [email-masked]
> > It needs more than rewording - it needs actual requirements. "MAY" is a
> rather weak term, and in the sentence above, it only applies to the server.
> How does the client detect this optional behavior by its peer? How is it to
> modify its own behavior in response? What is the protocol's justification for
> making it optional versus required? As written, the statement gives no
> guidance and therefore will not lead to interoperability.
> 
> these optional commands in the main NFSv4.2 spec.  The question is just how
> to best word the reference to it.  I agree that the MAY is not very specific,
> and I'm open to rewording it.  In fact I suspect thay the point has come
> where I made the intent clear and should leave the wording to professional
> spec wranglers..

Ok, fair enough. But the requirement for a DS serving a layout is different from a full-blown server. And tossing a downlevel requirement (4.1 DS from a 4.2 MDS?) makes it even more different. These need to be stated.

Frankly, it might be easier to consider the files layout to be a full-blown protocol, similar to how the blocks and objects layouts are currently handled. Iterating 4.x, as "x" rises above 2, is going to get hairy. Fixing it now could remove a lot of future confusion.

Just who are the "professional spec wranglers" you refer to btw? :-)

Tom.

---

From: Christoph Hellwig <[email-masked]>
Date: Mon, 10 Feb 2014 11:36:16 +0100

> Frankly, it might be easier to consider the files layout to be a full-blown protocol, similar to how the blocks and objects layouts are currently handled. Iterating 4.x, as "x" rises above 2, is go…

I tend to agree, but I have no idea how to do this in the framework of the
various published specs.

> Just who are the "professional spec wranglers" you refer to btw? :-)

Anyone who spent half a decade editing rfcs would qualify I think :)