Subject: [nfsv4] LAYOUT_STATS for NFSv4.2

From: Tom Haynes <[email-masked]>
Date: Tue, 29 Apr 2014 16:49:05 -0700

In the Flexible Files Layout, we are proposing that a client can inform
the metadata server about client I/O to the storage devices on a
layout return (see pnfs_ff_iostats4 in Sections 7.3 and 7.4 of
http://www.ietf.org/id/draft-bhalevy-nfsv4-flex-files-02.txt).

That is a very expensive operation because if the file is very hot,
returning the layout slows down the client.

We could also override LAYOUTCOMMIT to send back that information,
but I’d really prefer not to change the semantics of that operation to
deal with a statistics update.

What I would prefer is a new operation - LAYOUT_STATS. The client
could use that to inform the server as to a busy layout segment. That
would allow the server to do whatever it chose to do in order to optimize
data storage location.

I’ve avoided a callback in order to support a very simple push model.

I’ve also avoided any mandate as to how often the client MUST update
and on which files it MUST report.  And also, we don’t care as to how
the MDS acts on this information. All of these are implementation details.

Finally, this operation is mandatory.

Comments?

Here is the proposed set of changes to support this new operation:

---

From: Tom Haynes <[email-masked]>
Date: Tue, 29 Apr 2014 16:50:48 -0700

Whoops, sent this from the wrong email account. (I blame
Mail.app!)

Please consider it as from a Primary Data employee.

On Apr 29, 2014, at 4:49 PM, Tom Haynes <[email-masked]> wrote:

> <layout_stats.diff>

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 29 Apr 2014 20:51:54 -0400 (EDT)

Hi,

I think it would be desirable to do a round of future-proofing on 
LAYOUT* operations.

One thing we have found to be problematic for LAYOUT extension (and
that we've mentioned at least once in WG meeting) is that LAYOUTGET
lacks opaque data which could be overridden by new layout types.
(That's off topic, but it would still be desirable to fix--for example,
we needed to overload iomode to prototype metastripe parameters.)

The idea to permit clients to push statistics when convenient seems
definitely useful.  What I think might be desirable would be to generalize
the operations, whenever possible.  For example, the stats operation
could be generalized to "simple push typed data" and the typed
data defined as particular statistics in flex files.

Thoughts on that?

Regards,

Matt

----- "Tom Haynes" <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 29 Apr 2014 20:20:42 -0700

Whoops, missing some text:

 <section toc='exclude' title="DESCRIPTION">
   <t>
     The LAYOUT_STATS operation reports to the metadata server
     statistical information about client I/O with the storage
     devices over the the layout represented by the current
     filehandle, client ID (derived from the session ID in the
     preceding SEQUENCE operation), byte-range (ls_offset +
     ls_length), and ls_stateid.  ls_duration represents the number
     of seconds the reported burst of I/O lasted.  ls_rd_count,
     ls_rd_bytes, ls_wr_count, and ls_wr_bytes represent, respectively,
     the number of contiguous read and write I/Os and the respective
     aggregate number of bytes transferred within the reported
     byte range.
   </t>

   <t>
     As the data transfers are performed directly between the
     client and the storage devices, unless there exists a control
     protocol between the metadata server and the storage devices,
     such statistical data might not be gatherable to the metadata
     server.  LAYOUT_STATS MAY be used by the client to report I/O
     statistics back to the metadata server.  Since it is infeasible
     for the client to report every I/O that used the layout, the
     client MAY identify "hot" byte ranges for which to report I/O
     statistics.  The definition and/or configuration mechanism
     of what is considered "hot" and the size of the reported byte
     range is out of the scope of this document. The metadata
     server is then free to use this information about client I/O
     to optimize the data storage location.
   </t>
 </section>

Or as an alternative, we could modify IO_ADVISE:

loghyr:NFSv4.2 loghyr$ git diff nfsv42_middle_op_io_advise.xml
diff --git a/nfsv42_middle_op_io_advise.xml b/nfsv42_middle_op_io_advise.xml
index d1accae..6d826d3 100644
--- a/nfsv42_middle_op_io_advise.xml
+++ b/nfsv42_middle_op_io_advise.xml
@@ -142,16 +142,25 @@
   <section toc='exclude' title="IO_ADVISE4_INIT_PROXIMITY">
     <t>
       The IO_ADVISE4_INIT_PROXIMITY hint is non-posix in origin and
-      conveys that the client has recently accessed the byte range in
-      its own cache. I.e., it has not accessed it on the server, but
-      it has locally.  When the server reaches
-      resource exhaustion, knowing which data is more important allows the
-      server to make better choices about which data to, for example purge
-      from a cache, or move to secondary storage.  It also informs the
-      server which delegations are more important, since if delegations are
-      working correctly, once delegated to a client and the client
-      has read the content for that byte range, a server might never
-      receive another read request for that byte range.
+      can be used to convey that the client has recently accessed
+      the byte range in its own cache. I.e., it has not accessed
+      it on the server, but it has locally.  When the server reaches
+      resource exhaustion, knowing which data is more important
+      allows the server to make better choices about which data to,
+      for example purge from a cache, or move to secondary storage.
+      It also informs the server which delegations are more important,
+      since if delegations are working correctly, once delegated
+      to a client and the client has read the content for that byte
+      range, a server might never receive another read request for
+      that byte range.
+    </t>
+
+    <t>
+      The IO_ADVISE4_INIT_PROXIMITY hint can also be used in a pNFS
+      setting to let the client inform the metadata server as to
+      the I/O statistics between the client and the storage devices.
+      The metadata server is then free to use this information
+      about client I/O to optimize the data storage location.
     </t>
 
     <t>

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 29 Apr 2014 20:23:49 -0700

Hi Matt,

I haven't seen any proposed text.

Thanks,
Tom

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 30 Apr 2014 12:20:06 +0200

> the MDS acts on this information. All of these are implementation details.

I don't really like the model at all.  It's very vague and almost assumes
a closely cooperating server and client.  Also gathering your stats on
the server side is just as easy as on the client side, and a private
side channel protocol between those allows them to evolve based on the
needs of the specific architecture.

I do like your changes to IO_ADVISE, though - that is highlevel enough
to be useful for all kinds of different architectures, and imposes
very little constraints on the clients.

> Finally, this operation is mandatory.

With that you mean REQUIRED?

That contradicts with the minor versioning rules:

"15.  Except for infrastructural changes, a minor version must not
      introduce REQUIRED new features."

---

From: Benny Halevy <[email-masked]>
Date: Wed, 30 Apr 2014 16:09:31 +0300

> the server side is just as easy as on the client side, and a private

By server do you mean the Data Server?

>       introduce REQUIRED new features."

pNFS as a whole is optional, but if implemented we defined a subset
of the operations to be REQUIRED within the feature.
See Section 17 in RFC5661.

In any case, Tom, does it have to be REQUIRED?
How about servers that don't care about the layout_stats?

Benny

---

From: Benny Halevy <[email-masked]>
Date: Wed, 30 Apr 2014 16:11:53 +0300

> diff --git a/nfsv42_middle_op_mandlist.xml b/nfsv42_middle_op_mandlist.xml
> index e78a671..41d687e 100644
> --- a/nfsv42_middle_op_mandlist.xml
> +++ b/nfsv42_middle_op_mandlist.xml
> @@ -144,6 +144,7 @@
>        <c>LAYOUTCOMMIT         </c> <c> OPT </c> <c> pNFS (REQ)   </c>
>        <c>LAYOUTGET            </c> <c> OPT </c> <c> pNFS (REQ)   </c>
>        <c>LAYOUTRETURN         </c> <c> OPT </c> <c> pNFS (REQ)   </c>
> +      <c>LAYOUT_STATUS        </c> <c> OPT </c> <c> pNFS (OPT)   </c>

s/STATUS/STATS/

I'm OK with pNFS (OPT)

Benny

>        <c>LINK                 </c> <c> OPT </c> <c>              </c>
>        <c>LOCK                 </c> <c> REQ </c> <c>              </c>
>        <c>LOCKT                </c> <c> REQ </c> <c>              </c>

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 30 Apr 2014 08:42:17 -0700

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 30 Apr 2014 08:43:25 -0700

> With that you mean REQUIRED?

Whoops, I mean't to put a *not* there. Or optional. :-)

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 30 Apr 2014 18:29:29 +0200

> By server do you mean the Data Server?

For the files layout that would be the DS, for block and objects it would
be the storage target.  Note that any sufficiently sophisticated iSCSI or
FC target already tracks pretty similar statistics that the MDS could
query.

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 30 Apr 2014 10:22:09 -0700

On Apr 30, 2014, at 9:29 AM, Christoph Hellwig <[email-masked]> wrote:

>>> a closely cooperating server and client.  

Yes, it is vague - trying to abstract a specific need does that.

pNFS is built on either closely cooperating MDS and DS or
MDS and client. Look at how fencing is defined in the non-File
Layout Types - the client has to do some of the work.

>>> Also gathering your stats on

The model the Flexible File Layout Type is pursuing has both a loosely
and tightly coupled semantics. The main difference is the richness of
the "control protocol".

For the loosely coupled model, the DS can be NFSv3, NFSv4, NFSv4.1,
NFSv4.2, etc. But there is no communication path between the MDS and
the DSes other than NFS.

For the tightly coupled model, the DSes would be NFSv4.1 and NFSv4.2,
etc. The MDS would be able to communicate with the DSes over some
control protocol.

So, we'd like a model that does not assume any coupling between the
MDS and DS. If it is there, the MDS can use it.

I think IO_ADVISE is a better home than LAYOUT_STATS. While LAYOUT_STATS
makes it clear for the purpose, I'd rather fold it in where it is
more natural.

But I'd still like to allow the client to be able to inform the MDS about
the intensity of the I/O. I.e., right now we can say things like:

IO_ADVISE4_RANDOM | IO_ADVISE4_READ | IO_ADVISE4_INIT_PROXIMITY

the client is doing random reads and would like better access to the data.

But we can't help the MDS decide how important that data is to the client.

The choices Flex Files have:

1) is to only send the IO_ADVISE4_INIT_PROXIMITY when the client detects that
the I/O access has passed some threshold

2) add the following to the IO_ADVISE4args:

    iar_duration represents the number
    of seconds the reported burst of I/O lasted.  iar_rd_count,
    iar_rd_bytes, iar_wr_count, and iar_wr_bytes represent, respectively,
    the number of contiguous read and write I/Os and the respective
    aggregate number of bytes transferred within the reported
    byte range.

I.e., is it the client or the MDS which makes the call as to when a
threshold has been passed. And since it is a hint, the additional
fields need not be sent nor does the MDS have to take any action.

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 30 Apr 2014 20:15:55 +0200

> MDS and DS. If it is there, the MDS can use it.

So you say you want to be able to use a legacy NFS server aѕ a DS and
instead shift more work onto the client.  I don't really like this
and would prefer that hacks needed for that model would stay in the
flexible files layout specification instead of in the main NFS spec.

> the intensity of the I/O. I.e., right now we can say things like:

Right now you just shift over statistics to the MDS.  Can you instead
come up with something like an read and write intensity level that
could fit into the hints model?  If not adding it to IO_ADVISE seems
wrong.

If you really need the statistics of the past I/O doing it in the flex
files specific LAYOUTCOMMIT payload seems the most sensible option to me
if you mostly care about writes.  Of course it doesn't really apply
to reads.

The second best option would be to add the new command, even if I really
don't like the smell of it.

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 30 Apr 2014 11:34:44 -0700

> flexible files layout specification instead of in the main NFS spec.

I'd love to do it there. I just don't want to have to do a LAYOUTRETURN
to get the data. Yuck!

> could fit into the hints model?  

This is what I would prefer. Struggling with how to abstract it. :-)

I'll take some time to explore this out some more. When I thought
I was adding a new operation, I wanted to get the dialog started as
early as possible. A modification to the hints model is not as
pressing a matter.

> If not adding it to IO_ADVISE seems
> wrong.
> 

Right and I really do not want to make changes to allow LAYOUTCOMMIT
to report on READs.

Nor do I want to see LAYOUTCOMMIT and IO_ADVISE in the same compound
to get the data across. Now that is a hack!

> don't like the smell of it.

Yeah, I don't like it either now.

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Wed, 30 Apr 2014 15:57:47 -0400 (EDT)

Hi,

> > flexible files layout specification instead of in the main NFS spec.

That's exactly right.

> I'd love to do it there. I just don't want to have to do a

Right...

> to get the data across. Now that is a hack!

I don't understand the aversion to LAYOUTCOMMIT.  This is a well abstracted
operation that has opaque data.  It's semantics -should- be permitted
to vary by layout type.  I don't see why a layout type's LAYOUTCOMMIT can't
have variant semantics, to boot.

Matt

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 30 Apr 2014 13:45:47 -0700

Hi Matt,

Besides the fact that it is for WRITEs?

I would have loved for it to be LAYOUTUPDATE and have a field to indicate
that it was for READs and WRITEs. But wishes and fishes...

Anyway, from 5661, Section 12.5.4:

   For file-based layouts, synchronization of
   attributes between the metadata and storage devices, primarily the
   size attribute, is required.

I guess we could set loca_last_write_offset and loca_time_modify both to
0 to indicate that this was a READ update.

Or the flex files layout type could define structure in the opaque field with
an indicator about whether the LAYOUTCOMMIT meant normal
semantics or layout type specific semantics.

But I'd rather not open that can of worms.

Feel free to try to convince me otherwise!

Thanks,
Tom