Subject: Re: [nfsv4] New Version Notification for draft-bhalevy-nfs-obj-00.txt

From: "Matt W. Benjamin" <[email-masked]>
Date: Sun, 7 Oct 2012 20:37:11 -0400 (EDT)

Hi Tom,

----- "Tom Haynes" <[email-masked]> wrote:

> 4.2.3.  NFS Device Identifier The PNFS_OBJ_NFS pnfs_obj_type4 is used
> to identify NFS filer
>    devices.  In this case, ona_version and ona_minorversion represent
>    the NFS protocol version to be used to access the NFS filer. 
> Either
>    the "ona_netaddrs" or "ona_fqdn" fields are used to locate the
>    device. "ona_netaddrs" MAY be set to a list holding one or more of
>    the device network addresses.  Alternatively, "ona_fqdn" MAY be set
>    to the NFS device fully qualified domain name that can be resolved
> by
>    the client to locate the NFS device network address (See Domain
> Names
>    [22]).  The server MUST provide either one of "ona_netaddrs" or
>    "ona_fqdn", but it MUST NOT provide both.
> 
>    "ona_path" MUST be set by the server to an exported path on the
>    device.  The path provided MUST exist and be accessible to the
>    client.  If the path does not exist, the client MUST ignore this
>    device information and any layouts referring to the respective
>    deviceid until valid device information is acquired.
> 
> 
> I think this is an end run (*) over LAYOUT4_NFSV4_1_FILES.

This doesn't seem like a fair characterization.  The draft is seeking to do something -new- using the well-defined NFS (v3, v4, v41) protocols as raw materials. 
This appears to be a very efficient use of already-standardized bits of protocol.

It seems, on the other hand, like overreaching to propose that there can be only one standardized layout type which uses NFS-style RPCs as a transport.  It's just too convenient.  Meanwhile, the converse assumption leads to pressure from implementers to try to standardize new types of layouts for storage systems like Ceph, RxOSD, etc, which have many similarities to existing object storage systems, but whose native access protocol is not T10.  The already-standardized objects layout might be well-suited for several of these systems, if sufficient flexibility to use alternate access protocols were considered.  Again, the systems we are interested in appear to be much better suited to objects, than to files. There are many differences between the files layout and objects, that have nothing to do with the wire protocol (obviously).  We discussed several of these as they affect our Ceph implementation, at IETF 84.

>  
> 
> 
> If I were a customer, I would be nervous about the certification
> method used here. OSD components
> fall naturally into the Block Based Layout of RFC 5664 because they
> are basically disks, not
> filesystems. These NFS mounts are filesystems and questions are easy
> to see about directory
> consistency, file consistency, filesystem consistency, etc.

While Benny's written preface clearly indicates his interest in working with systems speaking NFSv3 or NFSv40, we have no interest in this, and do not see how our sub-dialect of the draft protocol would raise any of the issues related to mounting ordinary NFS filesystems.

When used with NFSv41 protocol operations (at least), and following the consistency mechanisms provided in the objects draft, it is not clear to me where any doubt about consistency arises.  The client may perform operations only under the auspices of a layout.  The consistency model the client must observe is well defined.  I see no justification for assuming any issues of file or filesystem consistency, unless the same objects are explicitly made available to be mounted as ordinary NFS filesystems.  Again, this need not be the case.  What am I missing?

> 
> 
> From a standards viewpoint, this is no longer work in the realm of
> OSD. The title is misleading
> and a client which understands LAYOUT4_OBJECTS_V2 would be able to
> access a layout
> which could be comprised solely of NFS devices. Where is the beef?

The beef is in the distinction between NFS the file access protocol, and use of a subset of NFS as a sub-protocol for access to OSDs.  

> 
> 
> I would be perfectly fine with a proposed layout which allowed the use
> of NFS exported paths as
> targets. But I am not fine with this proposal.
> 
> 
> By the way, fine with such a proposal does not mean I think it would
> be wise to implement. These
> NFS devices presented here have to be very dumb because of the lack of
> a control protocol between
> them and the MDS. I don't see how they handle locks, stateids, and any
> state in general. Once we
> throw in sparse files in NFSv4.2, I don't see how they would be
> communicated.

I don't follow how these data servers are obligated to handle locks, stateids, or state in general, any differently than ordinary objects data servers.

Again, while the draft as written clearly -allows- an implementation to nominate a bunch of ordinary (and wrt cluster membership, completely oblivious) NFS servers/paths as data servers, it just as clearly allows those data servers to be drawn exclusively from servers directly participating in a rich cluster environment, and hence, using a fully sufficient control protocol unknown to the NFS client.  The design of pNFS, it has always seemed clear to me, was fully intended to support such configurations.  The above criticism applies, obviously, only to an architecture which -actually does- use dumb, oblivious data servers.  It certainly doesn't apply to us.

> 
> 
> In short, these NFS devices are pretty dumb and do not give the
> benefits of a DS.
> 
> 
> And Section 13.5 makes it pretty clear that there is no control
> protocol.
> 
> 
> ---
> 
> 
> (*) For those whom American English slang might be confusing, please
> see http://www.thefreedictionary.com/end-run 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Adam C. Emerson" <[email-masked]>
Date: Mon, 08 Oct 2012 15:57:21 -0400

Salutations,

> ----- "Tom Haynes" <[email-masked]> wrote:
[snip]
> > throw in sparse files in NFSv4.2, I don't see how they would be
[snip]

As Matt mentioned, the draft as written certainly supports dumb NFS
servers with no knowledge that they are participating as data servers,
but does not seem to outlaw a control protocol.  As to sparse files,
the description of READ_PLUS in the NFSv4.2 draft says:

   If the data to be returned is comprised entirely of zeros, then the
   server may elect to return that data as a hole.  The server
   differentiates this to the client by setting di_allocated to TRUE
   in this case.  Note that in such a scenario, the server is not
   required to determine the full extent of the "hole" - it does not
   need to determine where the zeros start and end.

I read this to mean that even a relatively dumb data server that knows
it contains a sparse region can return this fact to the client without
having to worry about the content of other data servers, if the server
were to issue a layout with major version 4 and minor version 2.

We believe this layout type could be productively expanded with a
transport type for NFSv4.1 that indicated it should be treated as a
data server, including the communication of the stateid.  This would
combine the improved data server semantics with the more capable
description of the object layout, giving the best of both worlds.

We also believe that this layout type should be expanded with a
packing flag to allow it to support sparse as well as dense packing,
as sparse packing is superior for systems using irregular striping
patterns.

Respectfully yours,

Adam C. Emerson                                  <[email-masked]>

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 8 Oct 2012 20:28:42 +0000

Hey Matt,

My concerns are:

1) There are no use cases presented in the document. While what you want is valid,
it is not there and neither are the use cases Benny has in mind. For a first draft,
I expect to see motivation.

2) The use of NFS servers comes in from left field. The title has blocks and OSD
in it, not files. This is not motivated and with a lack of use cases, I do not see why
we need to mix things.

I.e., read this not as I am opposed to the idea, but that I believe it not to follow from
everything presented.

3) The text does not mandate a control protocol. 5661 does and 5664 IMHO assumes
it via the components only being accessible via OSD. The text allows for servers
to not be collocated. For example, I don't see how it handles fencing.

Thanks,
Tom

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 8 Oct 2012 20:46:07 +0000

Hi Adam!

> were to issue a layout with major version 4 and minor version 2.

Lets pick something that needs more communication - say Labeled NFS.
How does the MDS communicate the label to the DSes?

When it recalls the delegation, how is fencing supported against
client ops?

> description of the object layout, giving the best of both worlds.

I just don't see it right now in the draft as it is - not arguing against
it, just have concerns it is not being explored well enough.

One thing that bugs me about the draft is in figuring out when READ
means an OSD READ, a NFS READ, etc. This lack of clarity makes
it difficult to follow the flow of interaction with the MDS, DS as an OSD,
and DS as an NFS Server.

> as sparse packing is superior for systems using irregular striping

Bring that up with Benny.

> Adam C. Emerson                                  <[email-masked]>

And I hope I was respectful back!
Tom

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 8 Oct 2012 21:01:35 +0000

> to not be collocated. For example, I don't see how it handles fencing.

By the way, I meant collocated and not clustered.

One impact of this is that as it stands, nothing prevents a layout where one
component is at nfs.foo.com:/blah/blah and the other at nfs.bar.com:/wah/wah.

How would multi-domain access be handled here?

Hmm, I guess 5661 suffers from this as well.

---

From: Boaz Harrosh <[email-masked]>
Date: Tue, 9 Oct 2012 19:10:35 -0700

> as sparse packing is superior for systems using irregular striping

No! you do not need any "dense" or "sparse" crap with osd layouts.
Any kind of layout you might anticipate, including these two above,
are already easily describe-able by the objects layout today.

I am yet to see a layout pattern that is not supported with a
segment-wise pnfs-objects layout. All those in the world that I
see today, are. 

(OK all files or object based, block-device based are not because
 we do not have the extents list per component, (which we could))

Cheers
Boaz

---

From: Benny Halevy <[email-masked]>
Date: Thu, 11 Oct 2012 20:46:55 +0200

> it difficult to follow the flow of interaction with the MDS, DS as an OSD,

I'll try to clarify that in the next version.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 11 Oct 2012 21:45:16 +0200

> segment-wise pnfs-objects layout. All those in the world that I

Actually the objects layout does not provide a way to represent
the "sparse" packing pattern, striping is always "dense".

Benny

---

From: Boaz Harrosh <[email-masked]>
Date: Thu, 11 Oct 2012 14:02:27 -0700

> the "sparse" packing pattern, striping is always "dense".
You could with the mirror_cnt and missing_device_id

And also with small segments unit wide ..