Subject: [nfsv4] [NFSv4.2] Meeting Minutes for 09/08/2011

From: Thomas Haynes <[email-masked]>
Date: Thu, 8 Sep 2011 13:50:10 -0500

Attendees
--------------

- Tom Haynes (NetApp)
- Manjunath Shankararao (NetApp)
- Viswanath Bandi (NetApp)
- Dean Hildebrand (IBM)
- Trond Myklebust (NetApp)
- Mike Eisler (NetApp)
- Peter Staubach (EMC)
- Peter Honeyman (CITI)

+ IETF Note Well Agreement

   http://www.ietf.org/NOTEWELL.html

+ New draft format

+ WRITE_PLUS

Still no need yet.

+ INITIALIZE and max_hole_punch

We decided to drop the max_hole_punch attribute. A server that has limitations can
either decide to return a short write or be asynchronous.

+ IO_ADVISE

This was posted on 8/25, but we did not get an automatic notification.

It is available as: http://datatracker.ietf.org/doc/draft-hildebrand-nfsv4-ioadvise/

Mike will comment on sections 4.5 and 4.6.

+ end-to-end corruption checking

No real discussion here, will have to wait on the requirements to firm up on
the WG alias.

+ seek_hole and seek_data 

Trond brought up these up as LInux has introduced them as system calls and
Solaris already has them. They will likely become a POSIX standard.

The READ_PLUS implementation will allow the client to find the end of the hole,
but if in data, we will not be able to find the start of the next hole.

We decided to review the use cases (Trond to post some links) and scope out
how much effort to get this into 4.2.

---

From: "Mike Eisler" <[email-masked]>
Date: Fri, 9 Sep 2011 09:47:10 -0700

I meant to post this yesterday; apoloogies.

On Thu, September 8, 2011 11:50 am, Thomas Haynes wrote:

> It is available as:
> http://datatracker.ietf.org/doc/draft-hildebrand-nfsv4-ioadvise/
>

4.5. pNFS Considerations

The IO_ADVISE considerations for pNFS are very similar to the
COMMIT considerations for pNFS. That is, as with COMMIT,
some NFS server implementations prefer IO_ADVISE  be done
on the DS, and some prefer it be done on the MDS.

So for the file's layout type, it is proposed that NFSv4.2
include an additional hint:

   const NFL4_UFLG_MASK            = 0x0000003F;
   const NFL4_UFLG_DENSE           = 0x00000001;
   const NFL4_UFLG_COMMIT_THRU_MDS = 0x00000002;
   const NFL42_UFLG_IO_ADVISE_THRU_MDS
                                   = 0x00000004;
   const NFL4_UFLG_STRIPE_UNIT_SIZE_MASK
                                   = 0xFFFFFFC0;

   typedef uint32_t nfl_util4;

   enum filelayout_hint_care4 {
           NFLH4_CARE_DENSE        = NFL4_UFLG_DENSE,

           NFLH4_CARE_COMMIT_THRU_MDS
                                   = NFL4_UFLG_COMMIT_THRU_MDS,
           NFL42_CARE_IO_ADVISE_THRU_MDS
                                   = NFL42_UFLG_IO_ADVISE_THRU_MDS,

           NFLH4_CARE_STRIPE_UNIT_SIZE
                                   = 0x00000040,

           NFLH4_CARE_STRIPE_COUNT = 0x00000080
   };

The new hint is valid only on NFSv4.2 or higher. Any file's layout
obtained with NFSv4.1 MUST NOT have NFL42_UFLG_IO_ADVISE_THRU_MDS set. Any
file's layout obtained with NFSv4.2 MAY have NFL42_UFLG_IO_ADVISE_THRU_MDS
set. If the client does not implement IO_ADVISE, then it MUST ignore
NFL42_UFLG_IO_ADVISE_THRU_MDS.

If NFL42_UFLG_IO_ADVISE_THRU_MDS is set, then if the client implements
IO_ADVISE, then if it wants the DS to honor IO_ADVISE, the client MUST
send the operation to the MDS, and the server will communicate the advice
back each DS. If the client sends IO_ADVISE to the DS, then the server MAY
return NFS4ERR_NOTSUPP.

If NFL42_UFLG_IO_ADVISE_THRU_MDS is not set, then this indicates to client
that if wants to inform the server via IO_ADVISE of the client's intended
use of the file, then the client SHOULD send an IO_ADVISE to each DS.
While the client MAY always send IO_ADVISE to the MDS, if the server has
not set NFL42_UFLG_IO_ADVISE_THRU_MDS, the client should expect that such
an IO_ADVISE is futile. Note that a client SHOULD use the same set of
arguments on each IO_ADVISE sent to a DS for the same open file reference.
The server is not required to support difference advice for difference
DSes of the same open file reference.

4.6. Number of Supported File Segments

In theory IO_ADVISE allows a client and server to support multiple file
segments, meaning that different, possibly overlapping, byte ranges of the
same open file reference will support different hints. This is not
practical, and in general the server will support just one set of hints,
and these will apply to the entire file. However, there are some hints
that very ephemeral, and are essentially amount to one time instructions
to the NFS server, which will be forgotten momentarily after IO_ADVISE is
executed.

The following hints will always apply to the entire, regardless of the
specified byte range:

               IO_ADVISE4_NORMAL,
               IO_ADVISE4_SEQUENTIAL,
               IO_ADVISE4_SEQUENTIAL_BACKWARDS,
               IO_ADVISE4_RANDOM

The following hints will always apply to specified byte range, and will
treated as one time instructions:

               IO_ADVISE4_WILLNEED,
               IO_ADVISE4_WILLNEED_OPPORTUNISTIC,
               IO_ADVISE4_DONTNEED,
               IO_ADVISE4_NOREUSE

The following hints are modifiers to all other hints, and will apply to
the entire file and/or to a one time instruction on the specified byte
range:

               IO_ADVISE4_READ,
               IO_ADVISE4_WRITE

===============================

A different topic: IO_ADVISE4_RECENTLY_USED is not in Dean's IO_ADVISE I-D.

The argument raised in Quebec against IO_ADVISE4_RECENTLY_USED was
that this was effectively a hint about multi-level caching, and that
we should instead specify a more formal multi-level caching system.

Here's the problem I see with that: we've yet to specify a complete
(byte-granular_ single-level caching system. Indeed, I've an
internet-draft ( draft-eisler-nfsv4-pnfs-dedupe-01.txt) that illustrates
how complex full single-level caching is. I see no need to wait for
Nirvana when we have the opportunity, with a single bit,
IO_ADVISE4_RECENTLY_USED, to specify some very useful information to NFS
servers. Every NFS server vendor that implements two distinct classes of
storage knows what to do with IO_ADVISE4_RECENTLY_USED, and every client
that implements its own cache (single or multi-level) knows what to do
with IO_ADVISE4_RECENTLY_USED.
And as with any new feature of a minor revision of NFS4.2, no client and
no server is required to implement it.

For the benefit of those not at Quebec, the use case I discussed at the
meeting for IO_ADVISE4_RECENTLY_USED is that of the NFS client or
application restart. In the event of restart, the app's/client's cache
will be cold and it will need to fill it from the server. If the server is
maintaining a list (LRU most likely) of byte ranges tagged with
IO_ADVISE4_RECENTLY_USED, then the server could have stored the data in
these ranges into a storage medium that is less expensive than DRAM, and
faster than random access magnetic or optical media, such as flash. This
allows the end to end application to storage system to co-operate to meet
a service level agreement/objective contracted to the end user by the IT
provider.

I've articulated the benefit. I don't believe anyone has articulated the
harm.

---

From: Dean <[email-masked]>
Date: Mon, 26 Sep 2011 01:14:03 -0700

I posted a new version of the IO_ADVISE draft that includes the two 
sections from Mike below.  I also added a section for the 
IO_ADVISE4_RECENTLY_USED hint that includes Mike's text as well as some 
comments from the FADVISE draft.

http://tools.ietf.org/html/draft-hildebrand-nfsv4-ioadvise-01

Dean

On 9/9/11 9:47 AM, Mike Eisler wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4