Subject: [nfsv4] pnfs: hints for when to write where

From: Garth Goodson <[email-masked]>
Date: Mon, 10 Apr 2006 13:15:19 -0700

As some of us have been developing the Linux client, we have discovered 
that it may be useful to introduce an attribute that can help the client 
decide when to write directly to the (metadata server) MDS versus 
writing to the (data servers) DSs.  There seems to be some general 
agreement that a hint of this type might be useful.  However, we don't 
have a clear picture of what it might look like -- I'll describe a few 
possible scenarios and am looking for feedback.

The main use of this hint would be for the server to advise the client 
on when I/O should be issued to the MDS or to the DSs.  I/O to the MDS 
does not require getting a layout, or doing a layout commit.

The initial Linux client implementation assumed that small writes (below 
some threshold) should be done to the MDS without using a layout. 
However, there are cases when this is most likely not the best behavior. 
  For example, if the client already has a layout and has already 
written data that requires a layoutcommit, it isn't saving anything by 
doing the write to the MDS.  Some implementations may never store data 
on the MDS, in which case all writes to the MDS require extra hops to 
get to the correct location.  Some implementations may prefer to have 
small files exist solely on the MDS (avoiding the overhead of using 
layouts), while others may prefer to load balance them across the DSs 
(making better use of their caches).  This attribute should probably be 
queried at open time.

I would like to get some feedback to the following three questions:

1) is it worth having some form of attribute that communicates to the 
client server preferences (note, this would be a hint, it could not be 
enforced)?

2) what would this attribute look like?

3) is this attribute per file or per file system?

Assuming yes to (1) for the moment, (2) may be as simple as a 
size-threshold attribute that says writes below that size should go to 
the MDS, if the attribute is 0, no writes should go to the MDS, if the 
attribute is all 1s, all writes should go to the MDS.  Or it may be as 
simple as exporting the stripe-width and letting the client make its own 
decision.

We may want to make it slightly more complex by adding some type of 
range parameter.  This would enable things like: if the file is less 
than x bytes long, then write to the MDS, otherwise write to the DS.

Any thoughts, comments welcome.

-Garth

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 10 Apr 2006 15:49:32 -0500

> client server preferences (note, this would be a hint, it could not be 

Yes, it does seem logical and will certainly be helpful as
implementations and our understanding of them become more mature.

> 2) what would this attribute look like?

Not sure.

> 3) is this attribute per file or per file system?

Per file.

> than x bytes long, then write to the MDS, otherwise write to the DS.

Maybe a little more complex would be helpful as file contents change,
MDS policies change, etc.  We should assume that such an attribute
will likely change over the lifetime of the file.  THe MDS may change
its policies or they may change with the age of the file, etc.

While we have an attribute model that allows for later additions, would
it make sense to have an initial hint attribute that could be expanded
via a bitfield to allow for later additions.

Spencer

---

From: Marc Eshel <[email-masked]>
Date: Tue, 11 Apr 2006 19:18:44 -0700

> 3) is this attribute per file or per file system?

It should be both per file system and per file. We can start with the per 
file system if the per file is zero and when we have more information 
about the file we can switch to a per file size-threshold.

> decision.

This sound fine. If we will have the size-threshold for both file and file 
system than 0 for the file can mean look at the per file system value.
 
> than x bytes long, then write to the MDS, otherwise write to the DS.

I don't think this is needed.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 12 Apr 2006 14:22:59 +0200

> about the file we can switch to a per file size-threshold.

Presuming a common use case of creating, writing to the MDS, 
and closing small files in a couple RPCs and presuming that 
these files are rarely rewritten I think that a per-file 
system attribute is required since the server has no 
knowledge of the file size until it is first written.  In 
this use model if the client has dirty data for a file 
that's being closed (i.e. its file size is determined) and 
the file is smaller than or equal to the attr value then the 
client is advised to flush the data to the MDS. [Note that 
the client may already have a layout for the file if the 
client policy is to get a write layout when creating files 
opened for write]

If, for example, this becomes inefficient when the server is 
saturated, the server (or the admin) can change the 
filesystem attribute to direct more I/O at the DSs 
(hopefully managing layouts will put lighter weight on the 
MDS than proxying I/O for small files)

Another per-file attribute could be used for larger files 
that might be written "in-place" in such a pattern that 
could be more efficient written through the MDS rather than 
directly to the DSs.  For us this measure is related to the 
strip size which can be extracted from the layout so we 
wouldn't need another per-file attribute.

> system than 0 for the file can mean look at the per file system value.

I'm not sure everybody will agree to these semantics... How 
would you force I/O to the DSs for specific files?  My 
intuition on this matter is that if we have a per-file 
attribute it should be orthogonal to the per-fs attribute 
and not override it.

> I don't think this is needed.

Actually I think that a file size parameter makes some 
sense.  It can be used for "all or nothing" with the extreme 
  values you mentioned above.  If the attr is set to 0 then 
no  files (shorter than or equal to 0 bytes) will be written 
to the MDS, if it's set to ~0 then all files (shorter than 
or equal to ~0 bytes) will be written to the MDS.

---

From: Brent Welch <[email-masked]>
Date: Wed, 12 Apr 2006 09:07:36 -0700

(I've seen the whole thread.)

If the generic layers call out to the layout driver to make the
decision, then they can encapsulate whether it is a filesystem
attribute or a per-file attribute or some other logic about
if there is already a layout, etc. etc.  The spec has to define
the attributes, of course, but the generic layers should delegate
the way these are used to the layout driver.  This requires looking
at the LAYOUTTYPE attribute on the file to do the dispatch.

>>>Garth Goodson said:

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Garth Goodson <[email-masked]>
Date: Wed, 19 Apr 2006 16:44:04 -0700

Last week during the pNFS linux client weekly meeting we discussed an 
issue to do with the attribute being discussed (by this thread).  We 
came to the realization that if we are going to support a single file 
system exporting a single file through multiple different layout types, 
and if you want to give different hints for those layout types then the 
structure of the attribute gets slightly more complex.

I think most of us agreed that it would be worthwhile to be able to 
specify different values of the attribute for different layout types 
when multiple layout types are exported by the same file system.  If you 
buy into this then the simplest attribute would have to be an array of 
the 'base' attribute organized by layout type.

E.g., (pseudo xdr-code)

struct attrib_hint_value {
   pnfs_layout_type type;
   TBD;  /* real attribute hint value */
};

struct attrib_hint {
   attrib_hint_value values<>;
};

Although this does not help us come closer to actually defining the core 
attribute, I wanted to point this out.

-Garth

Garth Goodson wrote:
>  For example, if the client already has a layout and has already written 
> data that requires a layoutcommit, it isn't saving anything by doing the 
> write to the MDS.  Some implementations may never store data on the MDS, 
> in which case all writes to the MDS require extra hops to get to the 
> correct location.  Some implementations may prefer to have small files 
> exist solely on the MDS (avoiding the overhead of using layouts), while 
> others may prefer to load balance them across the DSs (making better use 
> of their caches).  This attribute should probably be queried at open time.
>