Subject: [nfsv4] List of possible work items for NFSv4.2

From: "Mike Eisler" <[email-masked]>
Date: Wed, 12 Aug 2009 20:41:55 -0700

At the Stockholm IETF meeting, I left with an
action item to list and summarize possible work
items for NFSv4.2 and provide this within two
weeks of the meeting.

Please read this list, and discuss it, and where
possible indicate your willingness to:

- contribute to NFSv4.2

- review NFSv4.2 proposals

- implement an NFSv4.2 client and/or server at
  supported one or more of these ideas.

And of course, suggest items to add to the list,
and items to remove from the list.

Work items apparently requiring a new minor version of NFSv4
============================================================

Peer-to-Peer NFS
----------------

See
http://tools.ietf.org/id/draft-myklebust-nfsv4-pnfs-backend-00.txt .

The proposal involves allowing pNFS clients to
become data servers for other pNFS clients thus
off loading the primary storage array.  As
presented at the Stockholm IETF meeting it
appears to be limited to read-only workloads
based on whole-file caching.  Feedback in
Stockholm was that read/write workloads should be
considered and that perhaps something along the
lines of
http://tools.ietf.org/html/draft-eisler-nfsv4-pnfs-dedupe-00
could be combined with the peer-to-peer NFS proposal to offer
sub-file caching. Note that the dedupe proposal
is easily extended to provide sub-file caching.
Regardless the concept was well received.

Given industry trends toward flash memory, and
that the economics of flash memory best realized
if the flash is close to the application,
Peer-to-Peer NFS with sub-file caching is a very
timely.

Copy
----

See http://tools.ietf.org/html/draft-lentini-nfsv4-server-side-copy-03 .

The proposal supports intra-NFS server and
inter-NFS server file copy.  This is not a new
idea for WG; it was mentioned in the BOF that
predated the formation of the WG over 10 years
ago. The idea has never had merit because there
have been now APIs on existing NFS clients to use
it. This is starting to change.  Hypervisors are
starting to have facilities (e.g.  VMware vSphere
system, see
http://www.vmware.com/files/pdf/key_features_vsphere.pdf)
that require the ability to copy storage between
storage devices. In addition, there is a proposal
to add a reflink() ( see
http://lwn.net/Articles/331576/ ) system call
that enables application to leverage a file
system's zero copy cloning capability.

There seems to be little controversy supporting
adopting file copy as a work item.

Note that as proposed, file copy requires a
revision to RPCSEC_GSS to enable a concept called
"privileges".  See
http://tools.ietf.org/html/draft-williams-rpcsecgssv3-00 .
While the original motivation for RPCSEC_GSSv3
was to support Mandatory Access Control, the
privileges concept has proven to be very useful
to express the type of delegated security
inter-server file copy requires.

Hole Punching
-------------

Regular files can contain holes: byte ranges that
are zero filled but do not contain allocated
space. Holes are useful for saving space.

Hole Punching is the act of zeroing out a byte
range of a file, and de-allocating the storage
for that byte range. This is easy to do in NFSv4
today if hole punching from the end of the file
downward: invoke SEATTR to reduce the file's
length, then invoke SEATTR to restore the file to
its original size.

Hope punching has been proposed for NFS in the
past. In the 1980s a draft proposal for NFSv3
included it. However, as with file copy, the lack
of APIs caused the idea to die.

What has changed is the existence of hypervisors.
A hypervisor virtualizes the storage of guest
operating systems. The guest thinks it is dealing
with a physical storage device and sends commands
to deallocate storage. The hypervisor intercepts
these requests. If using network storage, then
the hypervisor needs support in the storage
access protocol to deallocate blocks of storage.
I.e., it needs hole punching support.

Hole punching has had little (if not zero)
discussion on the NFSv4 mailing list, but it
would be inconsistent to adopt file copy without
hole punching.

MAC Security Attribute
----------------------

See http://tools.ietf.org/html/draft-quigley-nfsv4-sec-label-00.

The attribute is necessary to support Mandatory
Access Control.

This work has been presented several times at
IETF and has been discussed several times on the
mailing list. One issue is that there is no
consensus on defining REQUIRED
Domain-Of-Interpretations (DOIs) to ensure
interoperability, nor is there consensus that
such a thing is required. There is arguably a
precedent:  the mimetype attribute. The NFSv4.1
protocol does not explicitly define the content
of this attribute, relying on the standards
bodies that control mime types to define the
content. Mime types are similar to DOIs in the
action taken by the NFS client when reading the
type is not explained by the NFS protocol. Nor
are mandatory mime types defined. The difference
is that DOIs potentially have explicit
requirements on the NFSv4 server.

The idea of associating MAC with an
IANA-registered named attribute has been
suggested in the past, but according to the I-D
rejected because:

"[named attributes] lack a way to atomically set
 the attribute on creation.  In addition, named
 attributes themselves are file system objects
 which need to be assigned a security attribute.
 "

These are certainly issues. While we could
imagine changing the NFSv4.x protocol to allow
named attributes to be created atomically, more
thought would have to be put into what it means
to set what is effectively a permission attribute
on the permission attribute. Assigning special
semantics to one particular named attribute seems
to be what a RECOMMENDED or REQUIRED attribute
are designed to do.

It is clear (as evidenced by the energy behind
SELinux) that on the NFS client-side, especially
Linux, there is strong desire to support MAC. The
same level of desire does not appear to exist on
the NFS server-side, with several storage vendors
at IETF meetings indicating that in absence of a
customer demand, they would not be likely to
support the feature.

Traffic Classification
----------------------
Near the end of the feature freeze for NFSv4.1, a
proposal was made to specify priority channels.
This was very controversial and quickly
withdrawn. Nonetheless, the demand for
classifying or tagging streams of traffic never
goes away.

End-to-End Data Integrity
-------------------------
At various times during the NFSv4.1 standards
process, the topic of defining data integrity
checksum that would be kept in the storage device
and provided to the client when it read the data
was discussed. The motivation was to protect data
from silent corruption as it left the storage
media on read, or was sent from the client on
write.

Various issues were raised:
- the method by which this checksum was provided,
  as explicit NFSv4.x operations or via
  RPCSEC_GSS was controversial

- additional performance impact, especially if
  the client or server was already using
  integrity or privacy in RPCSEC_GSS (i.e. why
  calculate two different checksums)

- no support for operations other than READ and WRITE. I.e. metadata
  was not protected.

- how should mismatches between the alignment of
  transfer size of the client's I/O versus the
  server's on media check sum be handled?

- controversy as to whether this was a significant problem

The last issue is the key one. Without consensus
there is a problem to solve, this work item won't
go forward.

Umask Attribute
---------------
See http://www.ietf.org/proceedings/74/slides/nfsv4-3.pdf

The proposal is to include a umask attribute that
would be provided with the OPEN operation during
file creation.  This is not an attribute that
would be stored in the file but instead would
allow the NFSv4 client to indicate to the NFSv4
server what umask to apply to file when combining
the mode and/or acl attributes in the OPEN
arguments.

The proposal goes on to say that if there is a
default ACL on the file's directory, the server
can ignore the umask.  What is not explained is
what problem this solves, since the client could
combine the umask and mode on its side, and send
the OPEN with a mode attribute reflecting the
combination of umask and the mode asrgument to
the open() system call.

The proposal does say that if there is a default
ACL on the file's parent directory, the server
can ignore the umask. Apparently the purpose here
is to emulate a UNIX semantic that says that the
mode should be used as is when there is a default
ACL (but then how is the mode combined with any
corresponding user, group, and other ACEs in the
default ACL?).

More discussion is likely requireed for this
proposed item.

Shutdown Callback
-----------------
See http://www.ietf.org/proceedings/74/slides/nfsv4-3.pdf

The proposal is that the server will send a
callback in preparation for a planned shutdown.

The client can then react as needed: inform user,
unmount NFS file systems etc.

One reaction not mentioned is that the client
could commit modified data to the server.

This functionality replaces the "rwall" ONC RPC service.

Readahead Hint
--------------
See http://www.ietf.org/proceedings/74/slides/nfsv4-3.pdf

Today NFS servers use heuristics to determine if
a sequential read pattern exists, and if so, they
will schedule reads from their storage devices in
anticipation that by the time the client sends a
READ, the data will be in the server's cache.
This has drawbacks:

- With pNFS, a given storage device has
  difficulty detecting a read pattern, since the
  next logical block might be on the next
  device.

- NFS clients often have parallel threads issuing
  read requests. The pattern of READs as received
  by the server is not sequential.

- Detecting readahead requires a set of READs.

- For small files, the set of READs needed might
  exceed the length of the file

- The heuristics on the server can produce false
  positives.

It appears the proposal would consist of a new
operation that would be like READ, but would not
return data. Possible return values might be:

- requested ignored (server is too loaded)

- range is already in cache

- request in progress

pNFS Connectivity/Access Indication
-----------------------------------
See http://www.ietf.org/proceedings/75/slides/nfsv4-0.pdf, slides 112-121.

The issue is that a pNFS client might be able to
reach a storage device identified in a layout,
due to a misconfiguration in the network or on
the pNFS server. Ease-of-use considerations
motivate a way for the pNFS client to communicate
the problem to the MDS.

This communication could be in the form of an
extension to LAYOUT_RETURN, or a new operation.

There seemed to be consensus at the Stockholm
meeting that we want to solve this.

Better Negotiation of Session Reply Cache Sizes
-----------------------------------------------
After the WG meeting in Stockholm there was
discussion around how to enable a replier on a
session to pre-allocate the necessary space
needed for the reply cache without over
provisioning. One idea discussed to add an
operation that limits the set of operations that
can be used on the session. For example, a client
might create a session used only for operations
with results that are never cached, such as READ,
READDIR, and another session used only for
operations that are invariably cached, such as
WRITE, RENAME, REMOVE, etc. One problem with this
approach is that the operation would be sent
after the session was created, making it too late
for the server to pre-allocate the optimal size
for its reply cache.

Work items apparently not requiring a new minor version of NFSv4
================================================================

Metadata Striping
-----------------

See http://www.ietf.org/proceedings/73/slides/nfsv4-3.pdf .

The proposal is extend pNFS via a new layout type
to support distribution of metadata in a pNFS
server. A second type of MDS, the lMDS is
described. A pNFS client would be directed to an
lMDS via a layout returned by LAYOUTGET on the
new layout type. As proposed, only a new layout
type is needed.

The proposal has had little discussion on mailing
list, other than to clarify some points. At the
Minneapolis IETF meeting, it was noted that the
registered algorithms used for distributing
metadata by file name needed to be small in
number if pNFS clients were going to successfully
interoperate with any pNFS server.

De-Dupe Awareness and Sub-File Caching
--------------------------------------

See http://www.ietf.org/proceedings/73/slides/nfsv4-3.pdf .

The proposal is that NFS servers that support
space efficiency (i.e. data is the same between
two files is stored once), provide the space
efficiency maps to the NFS client. The maps are
encoded as bit maps, each bit corresponding to a
fixed sized block of a file.

The proposal does not require a new minor version
of NFS, but instead requires 64 new pNFS layout
types.

The proposal can be extended to support sub-file
caching, whether the file has de-duplication or
not, and is a candidate for marrying with the
peer-to-peer NFS proposal.

At the San Francisco and Minneapolis IETF
meetings, the feedback on the proposals has been
that block sizes and alignments that are powers
of 2 don't match up with all forms of
de-duplication and major use cases of caching.
For example suppose file 1 is 9111 bytes long and
file 2 is 1000 bytes long.  At offset 111, the
next 1000 bytes are equal to all of file 2. File
1 and file 2 are de-duplicated in some storage
devices.  A major use cache of caching that is
not covered by the proposal might be an HPC
application that has records are each aligned on
64 bit boundaries but with lengths that are not
powers of 2, e.g. the record lengths might be 108
bytes each (a multiple of 64 bits).

It seems obvious how the proposal could address
the HPC caching use case; simply relax the
requirement that block sizes be powers of 2.  More
thought will be needed to address unaligned
de-duplication use case, at least in its most
general forms.

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Tom Haynes <[email-masked]>
Date: Thu, 13 Aug 2009 08:33:59 -0500

> Work items apparently requiring a new minor version of NFSv4

Mike,

Is the intent to produce another huge document for 4.2 or to
have each of the work items reside in their own documents?

And then the 4.2 document would heavily reference the others
and be more of an overview?

Thanks,
Tom

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 13 Aug 2009 13:22:04 -0400

> possible indicate your willingness to:

Yes :-)

> - review NFSv4.2 proposals

Yes.

>   supported one or more of these ideas.

As stated at Stockholm, we set up a basic prototype of the peer-to-peer
NFS last summer. I hope to build that into a full implementation once
the 4.2 roadmap is clear.

This summer, I'm prototyping the FedFS proposal.

> and items to remove from the list.

I'd like to propose one minor item to speed up workloads that involve
creating lots of small files (e.g. 'untar'):

If the server hands out a write delegation when you open the file, I'd
like to relax the rule that it must synchronously create the file on
stable storage.
Instead, I propose that the server be allowed to defer actually storing
the file metadata until either the client sends a FILE_SYNC4 write, or
it sends a COMMIT, or the delegation is returned.

The reasoning is that as long as it holds the write delegation, the
client should always be able to fully recreate the file from scratch in
the unlikely event of a server reboot. It can do so since it knows that
the file has not been replaced, renamed or unlinked, nor has the data or
metadata been changed by some third party.

Cheers
  Trond

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Thu, 13 Aug 2009 13:51:00 -0400

> - review NFSv4.2 proposals

yes.

>  supported one or more of these ideas.

The striped meta-data sounds interesting.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Robert Gordon <[email-masked]>
Date: Thu, 13 Aug 2009 13:01:10 -0500

yes.

yes.

>  supported one or more of these ideas.

maybe..

Robert..

---

From: Rick Macklem <[email-masked]>
Date: Thu, 13 Aug 2009 14:12:21 -0400 (EDT)

> metadata been changed by some third party.
My understanding is that a server only recalls the delegation when
metadata changes that could affect access (such as mode, owner,
owner_group, acl) or size was changed. If that is correct, another
client could change some other attribute while the client holds the
write delegation.
- I think adding the additional requirement of "commit to stable storage"
   for such metadata changes, would be sufficient. (You could also
   exempt the client that holds the write delegation, so long as it
   understands that it might have to do the setattr again after a
   server restart, I think?)
--> Put another way, the server must either recall the delegation OR
   commit the new file to stable storage, whenever any Setattr is done
   by a client other than the one holding the delegation.

Does that sound right? rick

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 13 Aug 2009 14:30:30 -0400

> client could change some other attribute while the client holds the

Section 9.4 of RFC 3530 states that

        When a client has a read open delegation, it may not make any
        changes to the contents or attributes of the file but it is
        assured that no other client may do so.  When a client has a
        write open delegation, it may modify the file data since no
        other client will be accessing the file's data.  The client
        holding a write delegation may only affect file attributes which
        are intimately connected with the file data:  size, time_modify,
        change.

I've always read this as saying that the client can cache all attributes
aggressively when it holds a delegation of either type. Have I missed
something?

> Does that sound right? rick

If your interpretation of delegations is correct, then that sounds
reasonable.

One additional note: there is a case for allowing the NFS client to
unlink the file without ever forcing a commit to disk. It would allow
fast and efficient creation of nfs backed temporary files.

Cheers
  Trond

---

From: James Lentini <[email-masked]>
Date: Thu, 13 Aug 2009 14:30:58 -0400 (EDT)

>   supported one or more of these ideas.

I'd like to contribute in the above areas, especially with the 
copy-offload proposal.

-james

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 13 Aug 2009 13:32:42 -0500

I'd add this to the list:

 - capabilities

Volatile FHs come close, but there's no way to renew a stale volatile FH
other than by re-doing the LOOKUPs.  One fix would be to add an
operation for renewing volatile FHs and an error for indicating that a
volatile FH is "expired".

Alternatively: add capabilities as a separate concept.

Nico
--

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 13 Aug 2009 14:52:36 -0400

> it sends a COMMIT, or the delegation is returned.

	- Client creates a file with an open call returning a write
	  delegation.
	- Application does a stat, stores the inode number.
	- Server restarts, client resends the open.
	- Application does another stat, sees a different inode number.

Is that a problem?

I don't know if there are other attributes that might behave similarly.

Creation time, I guess.  And mode & ACL could be interesting if the
parent's inheritable ACLs change.

The solution to the latter might be to ensure delegation is returned
before the directory's permissions are changed.  Which I guess you'd
want to do anyway to assure the second creation wouldn't fail.

Are there races with lookup or readdir's from other clients?  Normally
if a client creates a file and then passes a message to another client,
that client could expect to see the results in readdir/lookup (its
applications might not, depending on client caching, but the client
would).

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 13 Aug 2009 15:26:53 -0400 (EDT)

> 	- Application does another stat, sees a different inode number.
Or (another issue w.r.t. being recreated with a different i-node #):
What happens w.r.t. the file handle? Such as:
 	- another client does a Lookup, GetFH of the file
 	- server crashes/reboots and then the client (re)creates the
 	  file.

"Might" be doable for volatile file handles, but another post in this
thread just mentioned the issue w.r.t. recovering volatile file handles.

rick

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 13 Aug 2009 15:36:34 -0400

> 	- Application does another stat, sees a different inode number.

If it cares, the client can force a COMMIT when the application first
calls stat. That being said, I'm not aware of too many applications out
there (other than backup applications) that care about tracking the
inode number.

> want to do anyway to assure the second creation wouldn't fail.

That seems a little tough on the client. Just have the server ensure
that it commits the directory contents before allowing the acl change.

> applications might not, depending on client caching, but the client

Do you know of an application that might use this? We could always have
the server deny readdir and negative lookups while in the grace period,
but that too seems like overkill. For instance I'm not aware that we
deny GETATTR calls today, despite the fact that some client may not yet
have reclaimed its write delegation, and so the server doesn't know to
whom it should direct the CB_GETATTR callback.

Cheers
   Trond

---

From: Rick Macklem <[email-masked]>
Date: Thu, 13 Aug 2009 15:42:06 -0400 (EDT)

> aggressively when it holds a delegation of either type. Have I missed
My understanding of this is that size, time_modify, change can be modified
locally in the client (since CBGETATTR allows the server to acquire 
them). Other attributes must be updated by a SETATTR to the server.

I'm thinking of when another client (not holding the delegation) does a
SETATTR of something like mimetype or hidden. I don't believe there is
anything in the RFC that says the server must recall a write delegation
for this. (It does have to recall a delegation for a SETATTR of owner,
owner_group, mode, acl from another client, since the client may no longer
be allowed to write to the file after the SETATTR is done.)

> fast and efficient creation of nfs backed temporary files.
This brings up the interesting case of numlinks, which is a read-only
attribute, but I don't think there is anything saying that the server
must recall a write delegation when another client does a Link Op.
--> So how can the client be sure that the unlink applies to the
     "last link" for the file?

At least that's how I understand things, which could certainly be
wrong, rick

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 13 Aug 2009 15:40:57 -0400

> thread just mentioned the issue w.r.t. recovering volatile file handles.

The second client can't access the file contents because the first
client created the file and still holds the write delegation. All it can
do is inspect the attributes, so why would it care if the file appears
to have been deleted and created afresh?

Cheers
  Trond

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 13 Aug 2009 15:48:49 -0400

> be allowed to write to the file after the SETATTR is done.)

See the above quote from rfc3530. It says other clients cannot make any
changes to the contents or attributes. I see no exceptions being made
for mimetype or hidden.

Another point is that changing these attributes should affect the value
of the change attribute. How is it then possible for the client with the
write delegation to cache that?

>      "last link" for the file?

See above 2 points.

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Thu, 13 Aug 2009 15:52:58 -0400 (EDT)

> to have been deleted and created afresh?
It cares because it gets a NFS4ERR_STALE and no longer knows how to
look the file up again (and doesn't expect to need to). I was assuming
non-volatile file handles were the seriously busted case, but the last
sentence might have suggested the opposite. (I'm assuming that the
fact the file isn't committed to stable storage meant that the same
applied to the i-node... which would normally be used to create a
non-volatile file handle. If you are committing everything but the
file's data at creation time, then I don't see how it differs from
what we already have?)

In summary, I'm saying that enough information needs to be committed
to stable storage, so that the file still has the same non-volatile
FH after the (re)create by the client done after a server reboot.

rick

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 13 Aug 2009 15:56:08 -0400

> FH after the (re)create by the client done after a server reboot.

...or the client could pass the filehandle and inode number to the
server when reclaiming the open during the grace period.

  Trond

---

From: Rick Macklem <[email-masked]>
Date: Thu, 13 Aug 2009 16:13:59 -0400 (EDT)

> changes to the contents or attributes. I see no exceptions being made
Ok, so I think you are assuming that the statement "assured that no
other client may do so" applies to all delegations, even though the
sentence is discussing read delegations. (I didn't read it that way.)

The sentences in the above quote w.r.t. write delegations makes no
mention of what other clients can or cannot do with attributes, it
only discusses what the client holding the write delegation can do
with the attribute.

> write delegation to cache that?
Good point. My reading of Sec. 9.4.3 is that the server maintains the
value of the change attribute, but uses the value returned by the
client with the write delegation to create an appropriate "change"
value. Since the server "sees" metadata changes from other clients,
it doesn't need to recall the write delegation in order to maintain
the "change" value.

But, I see that you can argue that the client holding the write
delegation will assume that "change" is not being changed by any
other client. As such, the server should recall the delegation
upon any SETATTR from another client. (I think that's what my
current server actually does, although I'd need to go look at
the sources to be sure.)

>>      "last link" for the file?
So, you are saying that a server must recall a write delegation when
another hard link to the file is created by any client?

Hmm, I don't think my server currently does that, but I can see the
argument for doing so, based on the above case of the client with
the write delegation not expecting the change attribute to be
changed by anyone else while it holds the write delegation.

I'd be interested in hearing what others believe is the correct
behaviour for this case? rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 13 Aug 2009 16:17:24 -0400

> that it commits the directory contents before allowing the acl change.

Sure.

> Do you know of an application that might use this?

No.

--b.

> We could always have the server deny readdir and negative lookups
> while in the grace period, but that too seems like overkill. For
> instance I'm not aware that we deny GETATTR calls today, despite the
> fact that some client may not yet have reclaimed its write delegation,
> and so the server doesn't know to whom it should direct the CB_GETATTR
> callback.

---

From: Tom Haynes <[email-masked]>
Date: Thu, 13 Aug 2009 17:26:18 -0500

> possible indicate your willingness to:

Yes

> - review NFSv4.2 proposals

Yes

>   supported one or more of these ideas.

I'm interested in either the Peer-to-Peer or Copy work.

But that has to factor into my development work on NFSv4.1. I.e., the 
NFSv4.2
work sounds interesting, but we need to deliver on NFSv4.1 first.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 13 Aug 2009 19:41:46 -0400

> > to stable storage, so that the file still has the same non-volatile
FH 
> > after the (re)create by the client done after a server reboot.

> server when reclaiming the open during the grace period.

That seems to add protocol complexity.

I think the better approach is to require the server to commit the new
file to stable storage if a client other that than one that has the
delegation looks it up.  If it gives that other client the filehandle
its committing (pun unintentional) itself to that filehandle and the
easiest way to do that is to commit the new file to stable storage.

I assume that this is going to be a really unusual case.  Normally, the
only client who will see this fh is the client with the delegation and
he would be prepared for the change.

I'm guessing that there are a few other case where committal of the file
has to be forced.  A READDIR that get fileid, for example.  

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, August 13, 2009 3:56 PM
To: Rick Macklem
Cc: J. Bruce Fields; [email-masked]
Subject: Re: [nfsv4] List of possible work items for NFSv4.2

On Thu, 2009-08-13 at 15:52 -0400, Rick Macklem wrote:
> >> "Might" be doable for volatile file handles, but another post in 
> >> this thread just mentioned the issue w.r.t. recovering volatile
file handles.
> > client created the file and still holds the write delegation. All it

> > can do is inspect the attributes, so why would it care if the file 
> > appears to have been deleted and created afresh?
> >

> to stable storage, so that the file still has the same non-volatile FH

> after the (re)create by the client done after a server reboot.

  Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 13 Aug 2009 19:43:21 -0400

I think we don't want a huge document but it isn't clear what
constitutes "huge".

For example, I think everybody is clear that the unchanged v4.1 stuff
will not be repeated in the v4.2 spec.  That alone should make it much
smaller than the v4.1 spec.

Beyond that, I don't know that anyone has thought about splitting things
up further.  I know I hadn't.  More documents means the possibility of
them disagreeing, although the IETF does like shorter documents.

As you indicate, there is going to have to be a v4.2 overview that will
describe any very small items as well as giving an overview.  Maybe we
start with a single v4.2 document and split things out if there is a
rough consensus that a particular item (or inter-related set of items)
makes sense in a separate document.  

I think the above applies to the base protocol.  Anything that is done
as part of a new layout type probably should be in its own document.  In
that case, the item may not really be part of v4.2.  There may be some
small feature in the base v4.2 to enable it, but in that case, it is
best to describe that separately, so that you don't assume it is used
only for the new layout type.

Other opinions?

-----Original Message-----
From: Tom Haynes [mailto:[email-masked]] 
Sent: Thursday, August 13, 2009 9:34 AM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] List of possible work items for NFSv4.2

Mike Eisler wrote:
> At the Stockholm IETF meeting, I left with an action item to list and 
> summarize possible work items for NFSv4.2 and provide this within two 
> weeks of the meeting.
>
> Please read this list, and discuss it, and where possible indicate 
> your willingness to:
>
> - contribute to NFSv4.2
>
> And of course, suggest items to add to the list, and items to remove 
> from the list.
>

Mike,

Is the intent to produce another huge document for 4.2 or to have each
of the work items reside in their own documents?

And then the 4.2 document would heavily reference the others and be more
of an overview?

Thanks,
Tom

---

From: "Muntz, Daniel" <[email-masked]>
Date: Thu, 13 Aug 2009 17:02:29 -0700

> From: Mike Eisler [mailto:[email-masked]] 
> Sent: Wednesday, August 12, 2009 8:42 PM
> To: [email-masked]
> Subject: [nfsv4] List of possible work items for NFSv4.2
> 
> 
> At the Stockholm IETF meeting, I left with an action item to 
> list and summarize possible work items for NFSv4.2 and 
> provide this within two weeks of the meeting.
> 
> Please read this list, and discuss it, and where possible 
> indicate your willingness to:
> 
> - contribute to NFSv4.2
> 

Yes.

> And of course, suggest items to add to the list, and items to 
> remove from the list.
> 

I've been thinking about whether we could improve performance of and/or
simplify GETATTR.  Now, it is one of the most frequently used
operations, but is one of the most complex to process.  Some
alternatives:

  1) If it can be determined that a particular attrmask is widely used,
create an operation specifically for retrieving that set of attributes.
The goal would be to improve performance by having the bulk of GETATTR
ops replaced by the new op, and that processing of the new op would be
faster than generic GETATTR handling.

  2) Split GETATTR into an op the retrieves fixed-size attributes, and
another op for variable-sized attributes.

  3) Choose an alternative to the current structure of GETATTR that
would allow rpcgen to generate XDR [en|de]code functions (e.g., an array
of attribute number/value pairs). 

> The proposal involves allowing pNFS clients to become data 
> servers for other pNFS clients thus off loading the primary 
> storage array.  As presented at the Stockholm IETF meeting it 
> appears to be limited to read-only workloads based on 
> whole-file caching.  Feedback in Stockholm was that 
> read/write workloads should be considered and that perhaps 
> something along the lines of 
> sub-file caching. Note that the dedupe proposal is easily 
> extended to provide sub-file caching.
> Given industry trends toward flash memory, and that the 
> economics of flash memory best realized if the flash is close 
> to the application, Peer-to-Peer NFS with sub-file caching is 
> a very timely.
> 
> Copy
> ----
> 
> See 
> http://tools.ietf.org/html/draft-lentini-nfsv4-server-side-copy-03 .
> 
> The proposal supports intra-NFS server and inter-NFS server 
> file copy.  This is not a new idea for WG; it was mentioned 
> in the BOF that predated the formation of the WG over 10 
> years ago. The idea has never had merit because there have 
> been now APIs on existing NFS clients to use it. This is 
> starting to change.  Hypervisors are starting to have 
> facilities (e.g.  VMware vSphere system, see
> that require the ability to copy storage between storage 
> devices. In addition, there is a proposal to add a reflink() 
> ( see http://lwn.net/Articles/331576/ ) system call that 
> enables application to leverage a file system's zero copy 
> cloning capability.
> 
> There seems to be little controversy supporting adopting file 
> copy as a work item.
> 
> Note that as proposed, file copy requires a revision to 
> RPCSEC_GSS to enable a concept called "privileges".  See 
> While the original motivation for RPCSEC_GSSv3 was to support 
> Mandatory Access Control, the privileges concept has proven 
> to be very useful to express the type of delegated security 
> Regular files can contain holes: byte ranges that are zero 
> filled but do not contain allocated space. Holes are useful 
> for saving space.
> 
> Hole Punching is the act of zeroing out a byte range of a 
> file, and de-allocating the storage for that byte range. This 
> is easy to do in NFSv4 today if hole punching from the end of the file
> downward: invoke SEATTR to reduce the file's length, then 
> invoke SEATTR to restore the file to its original size.
> 
> Hope punching has been proposed for NFS in the past. In the 
> 1980s a draft proposal for NFSv3 included it. However, as 
> with file copy, the lack of APIs caused the idea to die.
> 
> A hypervisor virtualizes the storage of guest operating 
> systems. The guest thinks it is dealing with a physical 
> storage device and sends commands to deallocate storage. The 
> hypervisor intercepts these requests. If using network 
> storage, then the hypervisor needs support in the storage 
> Hole punching has had little (if not zero) discussion on the 
> NFSv4 mailing list, but it would be inconsistent to adopt 
> file copy without hole punching.
> 
> MAC Security Attribute
> ----------------------
> 
> The attribute is necessary to support Mandatory Access Control.
> 
> This work has been presented several times at IETF and has 
> been discussed several times on the mailing list. One issue 
> is that there is no consensus on defining REQUIRED 
> Domain-Of-Interpretations (DOIs) to ensure interoperability, 
> nor is there consensus that such a thing is required. There 
> is arguably a
> precedent:  the mimetype attribute. The NFSv4.1 protocol does 
> not explicitly define the content of this attribute, relying 
> on the standards bodies that control mime types to define the 
> content. Mime types are similar to DOIs in the action taken 
> by the NFS client when reading the type is not explained by 
> the NFS protocol. Nor are mandatory mime types defined. The 
> difference is that DOIs potentially have explicit 
> IANA-registered named attribute has been suggested in the 
> past, but according to the I-D rejected because:
> 
> "[named attributes] lack a way to atomically set  the 
> attribute on creation.  In addition, named  attributes 
> themselves are file system objects  which need to be assigned 
> a security attribute.
>  "
> 
> These are certainly issues. While we could imagine changing 
> the NFSv4.x protocol to allow named attributes to be created 
> atomically, more thought would have to be put into what it 
> means to set what is effectively a permission attribute on 
> the permission attribute. Assigning special semantics to one 
> particular named attribute seems to be what a RECOMMENDED or 
> REQUIRED attribute are designed to do.
> 
> SELinux) that on the NFS client-side, especially Linux, there 
> is strong desire to support MAC. The same level of desire 
> does not appear to exist on the NFS server-side, with several 
> storage vendors at IETF meetings indicating that in absence 
> of a customer demand, they would not be likely to support the feature.
> 
> Traffic Classification
> ----------------------
> Near the end of the feature freeze for NFSv4.1, a proposal 
> was made to specify priority channels.
> classifying or tagging streams of traffic never goes away.
> 
> At various times during the NFSv4.1 standards process, the 
> topic of defining data integrity checksum that would be kept 
> in the storage device and provided to the client when it read 
> the data was discussed. The motivation was to protect data 
> from silent corruption as it left the storage media on read, 
> or was sent from the client on write.
> 
> The last issue is the key one. Without consensus there is a 
> problem to solve, this work item won't go forward.
> 
> Umask Attribute
> ---------------
> The proposal is to include a umask attribute that would be 
> provided with the OPEN operation during file creation.  This 
> is not an attribute that would be stored in the file but 
> instead would allow the NFSv4 client to indicate to the NFSv4 
> server what umask to apply to file when combining the mode 
> and/or acl attributes in the OPEN arguments.
> 
> The proposal goes on to say that if there is a default ACL on 
> the file's directory, the server can ignore the umask.  What 
> is not explained is what problem this solves, since the 
> client could combine the umask and mode on its side, and send 
> the OPEN with a mode attribute reflecting the combination of 
> umask and the mode asrgument to the open() system call.
> 
> The proposal does say that if there is a default ACL on the 
> file's parent directory, the server can ignore the umask. 
> Apparently the purpose here is to emulate a UNIX semantic 
> that says that the mode should be used as is when there is a 
> default ACL (but then how is the mode combined with any 
> corresponding user, group, and other ACEs in the default ACL?).
> 
> More discussion is likely requireed for this proposed item.
> 
> Shutdown Callback
> -----------------
> The proposal is that the server will send a callback in 
> preparation for a planned shutdown.
> 
> The client can then react as needed: inform user, unmount NFS 
> file systems etc.
> 
> One reaction not mentioned is that the client could commit 
> modified data to the server.
> 
> Today NFS servers use heuristics to determine if a sequential 
> read pattern exists, and if so, they will schedule reads from 
> their storage devices in anticipation that by the time the 
> client sends a READ, the data will be in the server's cache.
> This has drawbacks:
> 
> It appears the proposal would consist of a new operation that 
> would be like READ, but would not return data. Possible 
> return values might be:
> 
> The issue is that a pNFS client might be able to reach a 
> storage device identified in a layout, due to a 
> misconfiguration in the network or on the pNFS server. 
> Ease-of-use considerations motivate a way for the pNFS client 
> to communicate the problem to the MDS.
> 
> This communication could be in the form of an extension to 
> LAYOUT_RETURN, or a new operation.
> 
> There seemed to be consensus at the Stockholm meeting that we 
> want to solve this.
> 
> After the WG meeting in Stockholm there was discussion around 
> how to enable a replier on a session to pre-allocate the 
> necessary space needed for the reply cache without over 
> provisioning. One idea discussed to add an operation that 
> limits the set of operations that can be used on the session. 
> For example, a client might create a session used only for 
> operations with results that are never cached, such as READ, 
> READDIR, and another session used only for operations that 
> are invariably cached, such as WRITE, RENAME, REMOVE, etc. 
> One problem with this approach is that the operation would be 
> sent after the session was created, making it too late for 
> the server to pre-allocate the optimal size for its reply cache.
> 
> Work items apparently not requiring a new minor version of 
> NFSv4 ================================================================
> 
> Metadata Striping
> -----------------
> 
> The proposal is extend pNFS via a new layout type to support 
> distribution of metadata in a pNFS server. A second type of 
> MDS, the lMDS is described. A pNFS client would be directed 
> to an lMDS via a layout returned by LAYOUTGET on the new 
> layout type. As proposed, only a new layout type is needed.
> 
> The proposal has had little discussion on mailing list, other 
> than to clarify some points. At the Minneapolis IETF meeting, 
> it was noted that the registered algorithms used for 
> distributing metadata by file name needed to be small in 
> The proposal is that NFS servers that support space 
> efficiency (i.e. data is the same between two files is stored 
> once), provide the space efficiency maps to the NFS client. 
> The maps are encoded as bit maps, each bit corresponding to a 
> The proposal does not require a new minor version of NFS, but 
> instead requires 64 new pNFS layout types.
> 
> The proposal can be extended to support sub-file caching, 
> whether the file has de-duplication or not, and is a 
> candidate for marrying with the peer-to-peer NFS proposal.
> 
> At the San Francisco and Minneapolis IETF meetings, the 
> feedback on the proposals has been that block sizes and 
> alignments that are powers of 2 don't match up with all forms 
> of de-duplication and major use cases of caching.
> For example suppose file 1 is 9111 bytes long and file 2 is 
> 1000 bytes long.  At offset 111, the next 1000 bytes are 
> equal to all of file 2. File
> 1 and file 2 are de-duplicated in some storage devices.  A 
> major use cache of caching that is not covered by the 
> proposal might be an HPC application that has records are 
> each aligned on
> 64 bit boundaries but with lengths that are not powers of 2, 
> e.g. the record lengths might be 108 bytes each (a multiple 
> of 64 bits).
> 
> It seems obvious how the proposal could address the HPC 
> caching use case; simply relax the requirement that block 
> sizes be powers of 2.  More thought will be needed to address 
> unaligned de-duplication use case, at least in its most general forms.
> 
> 
> --

---

From: Robert Gordon <[email-masked]>
Date: Thu, 13 Aug 2009 19:10:43 -0500

> makes sense in a separate document.

This seems to make sense to me, one doc.

Robert.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 13 Aug 2009 21:11:59 -0400

> has to be forced.  A READDIR that get fileid, for example.  

Agreed. Bruce and I came to the same conclusion over a couple of beers
tonight :-)

I think that, as you state above, the case where more than one client
looks up the file is likely to be the exception rather than the rule.
Furthermore, it shouldn't really make much of a difference whether you
take the hit of having to commit the file to stable storage at OPEN
time, or at LOOKUP time, so being able to defer until you actually see
if someone else does a lookup will usually be a win...

Cheers
  Trond

---

From: Nick Williams <[email-masked]>
Date: Fri, 14 Aug 2009 10:13:33 +0100

Apologies if I'm off the mark - I'm not sure I fully grok yet the underlying
problem, but my suspicion is that there is a use-case where this will bite:
distributed compiles. I've seen people try and use things like distcc over
NFSv3 storage and frequently hit strange caching issues, where a remote node
tries to link with an object file just written by another node. The typical
workaround for this use-case is to turn off directory caching if the client
supports it. It would be nice if we could fix this problem in v4.

Nick.

On Fri, Aug 14, 2009 at 2:11 AM, Trond Myklebust <[email-masked]
> wrote:

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 14 Aug 2009 08:53:36 -0400

On Fri, 2009-08-14 at 10:13 +0100, Nick Williams wrote:
> Apologies if I'm off the mark - I'm not sure I fully grok yet the
> underlying problem, but my suspicion is that there is a use-case where
> this will bite: distributed compiles. I've seen people try and use
> things like distcc over NFSv3 storage and frequently hit strange
> caching issues, where a remote node tries to link with an object file
> just written by another node. The typical workaround for this use-case
> is to turn off directory caching if the client supports it. It would
> be nice if we could fix this problem in v4.

Distributed compiles are a case where you want to turn off delegations
so that you don't keep getting hit by the delegation recalls which occur
whenever the remote node tries to access the file. Those would bite you
far harder than any sync to disk issues.

IOW: I don't think this is a case which would really be affected either
way by my proposal. It needs a different solution.

Cheers
  Trond

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 14 Aug 2009 10:08:44 -0400

> way by my proposal. It needs a different solution.

Now that the caffeine is kicking in...

When you combine the peer-to-peer NFS proposal with the ability to cache
create on the server, then you get a system where the file doesn't need
to ever hit the disk, yet is accessible to all other clients by means of
peer-to-peer NFS. That might help speed up the distributed compilation
case (assuming that the pnfs overhead isn't too high...).

Until we get some form of writeable directory delegations, it won't
change the need to turn off directory lookup caching though.

Cheers
  Trond

---

From: "David P. Quigley" <[email-masked]>
Date: Fri, 14 Aug 2009 14:56:12 -0400

>   supported one or more of these ideas.

I'm willing to continue work on all of the above for the MAC security
attribute. I've also been thinking about some of the issues that you've
outlined in the description of it and it may be possible to reorganize
the DOI concept such that you can have a list of initial DOIs to address
your interop concerns.

Dave

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 14 Aug 2009 16:57:45 -0500

> the DOI concept such that you can have a list of initial DOIs to address

I'm willing to continue editing the RPCSEC_GSSv3 I-D.

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Sat, 15 Aug 2009 06:56:50 -0700

> and be more of an overview?

Tom,

Our Area Advisor has been unequivocal: no more huge documents.

There are at least three models that would meet the requirement Lars
has noted:

- A single document that describes differences between 4.2 and 4.1 (sans
  any new layout types). Conceivably, this document might be small enough
  (let's say the limit is 200 pages, as a data point,
http://www.rfc-editor.org/rfc/rfc5324.txt publish in July 2008, is 216
pages, and is the
  largest one I found published in the past 13 months) for
  IESG and the RFC editor to deal with. If not ...

- As you stated: a separate document for each work item and a master
  4.2 document that refers to the others.

- A set of 4.2 documents that lists the differences between 4.2 and 4.1.
  For example, we might have a document that lists all the new operations,
  as well as any operations that we've changed (or perhaps just what has
  changed within the operation);
  another document that lists all the new error codes, and
  provides complete tables that map errors to ops, and ops to errors; one
  document for each new layout type we decide to add; one document that
  lists the new and changed RECOMMENDED and REQUIRED attributes; etc.

I prefer the first model, but if there are things the WG wants in v4.2,
and if we can get them done as quickly as we did v4.1 (by any measure,
the amount of time we took to get v4.1 done, whether measured from when
v4.0 was done, or from when v4.1 was added as a work item, i.e. 3 to 5
years, was record pace for an NFS protocol turn), then let's use whatever
model keeps us under the 200 page limit per RFC. I suggest starting with
the first model, and then if hit 200 pages, debate then (and only then,
I don't see a need to cross a bridge until we get to it) how we want to
break the document apart.

(And to be clear, I said "let's say the limit is 200 pages" ... it might
be higher or lower. Even if it is higher, we might be better off setting
a lower limit, as smaller docs are easier to through the process.)
-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: "Mike Eisler" <[email-masked]>
Date: Sat, 15 Aug 2009 07:58:06 -0700

Reading through responses, both those published to the list and privately
sent to me, I see a strong consensus for NFSv4.2, strong indication
that there are people willing to contribute to the document process as
orginators, editors, and reviewers, and strong indication that this work
will be prototyped/implemented.

I propose:
- that NFSv4.2, RPCSEC_GSSv3, MAC be authorized as a work items immediately.
-- I further propose that the primary editors of these three work items be,
   respectively, Mike Eisler, Nico Williams, and Dave Quigley.
- For the NFSv4.2 document, we start by combining the COPY operations I-D,
  and the part of the MAC I-D that describes the RECOMMENDED MAC attribute.
-- We should have a discussion and reach consensus on how to combine
   sub-file caching
(http://tools.ietf.org/html/draft-eisler-nfsv4-pnfs-dedupe
   or
http://tools.ietf.org/html/draft-myklebust-nfsv4-byte-range-delegations),
   peer-to-peer NFS (
http://tools.ietf.org/html/draft-myklebust-nfsv4-pnfs-backend ),
   and unstable file create (see Trond's emails on this thread),
   and then fold this consensus into NFSv4.2.

Going through the items one by one:

> Peer-to-Peer NFS

Strong indication of support for this work in terms of reviewing,
contributing and implementing (it has already been prototyped).
Trond wants to re-do the prototype once the NFSv4.2 form of this
work is clarified.

> Copy

James indicated he'd like to continue to contribute here, including
with implementation. Tom Haynes also indicated a possibility to contribute
here.

> Hole Punching

I received a private response that proposals for hole punching and
thin provisioning will be coming soon.

> MAC Security Attribute

Dave Quigley is committed to contributing to

> Traffic Classification

Thus far, no renewed interested has been expressed on this.

> End-to-End Data Integrity

Thus far, no renewed interested has been expressed on this.

> Umask Attribute

Thus far, no renewed interested has been expressed on this.

> Shutdown Callback

Thus far, no renewed interested has been expressed on this.

> Readahead Hint

Thus far, no renewed interested has been expressed on this.

> pNFS Connectivity/Access Indication

A fresh item from Stockholm IETF meeting; absence of
re-stating support for this is not absence of proof of interest.
I for one am committed to seeing this solved in NFSv4.2.

> Better Negotiation of Session Reply Cache Sizes

A fresh item from after Stockholm IETF meeting; absence of
re-stating support for this is not absence of proof of interest.
But I remain luke warm on this ... not entirely convinced there is
a problem to solve.

> Metadata Striping

Andy Adamson has expressed interest in prototyping this.

> De-Dupe Awareness and Sub-File Caching

In the context of peer-to-peer NFS, I see continued interested
in sub-file caching. Sub-file caching as proposed my me, gives
one de-dupe awareness for free (or the converse depending on
how one wants to look at it). As I've proposed at the beginning of
this message, the WG needs to discuss how it wants to approach all
these topics before adding them to an NFSv4.2 I-D.

New items:

- unstable file creates. This has a lot of interested within the WG
- capabilities: An item proposed by Nico Williams
- attributes: Dan Muntz expressed an interest in revisiting how attributes
  are handled.

---

From: "David P. Quigley" <[email-masked]>
Date: Mon, 17 Aug 2009 10:06:46 -0400

I'm also committed to being a secondary author on the RPCSEC_GSSv3
document and I will be glad to work with anyone who wants to help author
a proper document on DOIs and Interoperability using the MAC attribute.

Dave

---

From: sfaibish <[email-masked]>
Date: Tue, 18 Aug 2009 09:23:45 -0400

Mike,

I was in vacation the last 2 weeks and I couldn't reply to your email until
now. I hope it is not too late.

>    respectively, Mike Eisler, Nico Williams, and Dave Quigley.
I will help with the review.

>    and then fold this consensus into NFSv4.2.
I will be part of the discussion. I would also be interested to combine
the sub-file caching for file, block and object layouts with some cross
mapping. I can explain in more detail if it is of interest. I would also
add to this perhaps an option to add later additional layout drivers. I  
will
be also interested to extend the P2P to take care of large numbers of small
files to address the explosion of number of objects scalability. I believe
that Benny Halevy will be also interested to contribute.

I want also to propose including a tighter error detection and reporting
related to pNFS access permissions I proposed at the meeting. I will extend
similar stats permission errors for the pNFS server.

One additional item should be the support of IPV6 in pNFS if there are any
gaps in the protocol. I will try to work with Chuck Lever on this item.

> Trond wants to re-do the prototype once the NFSv4.2 form of this
[Sorin] I will help with the review and test the prototype.

> thin provisioning will be coming soon.
[Sorin] I will contribute to this item including possible prototyping.

> I for one am committed to seeing this solved in NFSv4.2.
[Sorin] I will be involved in drafting the changes.

> Andy Adamson has expressed interest in prototyping this.
[Sorin] This relates close to the large number of small objects scalability
and I will work with Andy on this.

> these topics before adding them to an NFSv4.2 I-D.
[Sorin] I want to contribute to this.

> - attributes: Dan Muntz expressed an interest in revisiting how  

/Sorin

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Network Storage Group

         EMC²
where information lives

Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 18 Aug 2009 11:35:38 -0500

Thanks for pulling this together, Mike.

I will be working with Lars for updates to the charter.

Spencer

---

From: R N ALEX <[email-masked]>
Date: Tue, 18 Aug 2009 22:19:56 +0530

> gaps in the protocol. I will try to work with Chuck Lever on this item.

I would like to contribute to the IPv6 related work for NFS4.2 as
continuation of the general IPv6 discussions had as part of the 75th
stockholm IETF meeting.

---

From: "Sorin Faibish" <[email-masked]>
Date: Tue, 18 Aug 2009 19:49:53 -0400

On Tue, 18 Aug 2009 12:49:56 -0400, R N ALEX <[email-masked]> wrote:

>> > -- I further propose that the primary editors of these three work  
>> items
>> > be,
Alex, this is great. Welcome joining me and Chuck to discuss the gaps  
before
we start a draft.

        EMC²
where information lives

---

From: Mahesh Siddheshwar <[email-masked]>
Date: Wed, 19 Aug 2009 17:49:10 -0700

yes.
yes.
maybe..

Thanks,
Mahesh

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 20 Aug 2009 13:40:57 -0400

>   lists the new and changed RECOMMENDED and REQUIRED attributes; etc.

If they're going to be split up, the split should be logical enough to
allow a specialist to get away with ignoring some of the
documents--otherwise the review burden seems likely to be the same as it
would be for a single large document.  I think that rules out the third
option.

It would be nice to shorten the release cycle as well as the document
size.  The two reinforce each other: the total size of 4.2 may be hard
to control if participants are worried that this is the only chance to
get their feature in for X many years.

(In fact I think it would be best to keep to a roughly
one-feature-per-minorversion rule, and pick a set of minorversions to
work on simultaneously, thus preventing one slow-moving work item from
holding back others unnecessarily.)

--b.

---

From: Lisa Week <[email-masked]>
Date: Thu, 20 Aug 2009 11:55:04 -0600

Yes.

Yes.

>  supported one or more of these ideas.

Can't commit at this time (partly due to the amount of 4.1 still on  
our plates).

- Lisa

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 20 Aug 2009 16:18:52 -0400

> If they're going to be split up, the split 
> should be logical enough to allow a specialist 
> to get away with ignoring some of the
> documents

I guess you mean to get away with it, and not have a problem later come
back to bite us :-)  It's hard enough to get a logical division.  Even
harder to get a logical division and a proof that it has the desired
characteristic.  Hardest of all to have the proof's validity stand the
test of time.

> It would be nice to shorten the release cycle 
> as well as the document size.  The two reinforce 
> each other: the total size of 4.2 may be hard
> to control if participants are worried that this 
> is the only chance to get their feature in for X 
> many years.

I agree but my impression is that the IETF, as little as it likes large
documents, is not quite ready to say "OK.  Do v4.x for 1 < x <= n with n
<= 10, as it suits you."  If they did, I'd probably argue that one
feature per version was too small and we might wind up with n in the
range of 3 to 5.  Just a guess.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Thursday, August 20, 2009 1:41 PM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] List of possible work items for NFSv4.2

--b.

---

From: Tom Haynes <[email-masked]>
Date: Thu, 20 Aug 2009 15:47:08 -0500

> get their feature in for X many years.

Do you mean the release cycle in reference to just the document or with 
regards to implementations
as well?

A general concern with regards to the process is that we are just now 
implementing parts of
4.1 and may still need to get clarification in the 4.1 spec. So it is 
hard to focus on new features for
4.2 when we are still trying to get 4.1 right.

And as a community, we are finding parts of the mandatory parts that we 
are voting with our feet.
Can anyone say SSV?

I'm not saying we should wait until 4.1 is done before we start the 
requirements/design of 4.2.

I am saying that large documents lead to large implementation phases and 
it will be hard to
juggle multiple major (sorry, I mean *minor*) releases concurrently.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 21 Aug 2009 11:51:20 -0400

> regards to implementations

I'd think the two would go together: the larger the document, the longer
it takes to write, to implement, and to get out into the field.

I'd hope that smaller incremental steps would allow us to get more
implementation and user feedback sooner, and perhaps avoid the sort of
problems you describe with less in-demand (or harder-to-implement)
features riding the coattails of more "popular" features in a large
document.

> are voting with our feet.

(Though I'm sort of sad about SSV--are you saying you think it's dead,
or just delayed?)

--b.

---

From: Sam Falkner <[email-masked]>
Date: Mon, 24 Aug 2009 22:35:09 -0600

Yes.

Yes.

>  supported one or more of these ideas.

Possibly.

> Readahead Hint

I am willing to propose and revise text, and possibly prototype the  
three items above.

- Sam

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Aug 2009 15:26:22 -0500

An item that came up recently: nanosecond resolution timestamps (ctime,
mtime, atime), for POSIX.1-2008.

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 27 Aug 2009 13:51:43 -0700

> mtime, atime), for POSIX.1-2008.

NFSv4.[01] have nanosecond resolution.

3.3.1.  nfstime4

   struct nfstime4 {
           int64_t         seconds;
           uint32_t        nseconds;
   };

   The nfstime4 data type gives the number of seconds and nanoseconds
   since midnight or 0 hour January 1, 1970 Coordinated Universal Time
   (UTC).

So does NFSv3:

      struct nfstime3 {
         uint32   seconds;
         uint32   nseconds;
      };

   The nfstime3 structure gives the number of seconds and
   nanoseconds since midnight January 1, 1970 Greenwich Mean Time.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Aug 2009 15:54:43 -0500

> NFSv4.[01] have nanosecond resolution.

Yes, that's true.  I wasn't specific enough.  The issue is that the
actual filesystem might not have it, which means that
pathconf(_PC_TIMESTAMP_RESOLUTION) can't actually be implemented
properly for NFSv{3,4}.  We need a filesystem attribute for this.

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 27 Aug 2009 14:26:12 -0700

NFSv4.[01]:

5.8.2.41.  Attribute 51: time_delta

   Smallest useful server time granularity.

NFSv3:

      time_delta
         The server time granularity. When setting a file time
         using SETATTR, the server guarantees only to preserve
         times to this accuracy. If this is {0, 1}, the server
         can support nanosecond times, {0, 1000000} denotes
         millisecond precision, and {1, 0} indicates that times
         are accurate only to the nearest second.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 27 Aug 2009 17:31:04 -0400

> properly for NFSv{3,4}.  We need a filesystem attribute for this.

Why isn't the time_delta attribute sufficient? It is a per-filesystem
attribute that is supposed to return the "smallest useful server time
granularity" for that filesystem.

FSINFO's time_delta is supposed to provide the same information in
NFSv3.

Cheers
  Trond

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 27 Aug 2009 16:28:52 -0500

>    Smallest useful server time granularity.

Ah, thanks!

---

From: Tigran Mkrtchyan <[email-masked]>
Date: Fri, 28 Aug 2009 11:20:23 +0200

May be I am wrong, but from my point of view READDIR is one of the
places where some changes are required.

The POSIX API ( and cthon test ) expect that readdir call make's a snapshot
of a directory listing which valid until closedir is called. But there
are not hints for NFs server to invalidate this snapshot ( in other words
there is no CLOSEDIR operation in NFS ). The result of this is that
if you have huge directory ( let say 300k files ) which is rapidly changing
NFs server has a hard time to serve it, especially if clients are slow.

There was a couple of changes since v2, but all of them was related to
attributes delivered with listing.

Tigran.

 > - contribute to NFSv4.2
yes
 > - review NFSv4.2 proposals
yes
 >  supported one or more of these ideas.
yes

---

From: "Muntz, Daniel" <[email-masked]>
Date: Fri, 28 Aug 2009 04:17:46 -0700

I'll second this one.  I couldn't make up my mind which function was
annoying me more: readdir or getattr :-)  Let's look at both.

  -Dan

> From: Tigran Mkrtchyan [mailto:[email-masked]] 
> Sent: Friday, August 28, 2009 2:20 AM
> To: [email-masked]
> May be I am wrong, but from my point of view READDIR is one 
> of the places where some changes are required.
> 
> The POSIX API ( and cthon test ) expect that readdir call 
> make's a snapshot of a directory listing which valid until 
> closedir is called. But there are not hints for NFs server to 
> invalidate this snapshot ( in other words there is no 
> CLOSEDIR operation in NFS ). The result of this is that if 
> you have huge directory ( let say 300k files ) which is 
> rapidly changing NFs server has a hard time to serve it, 
> especially if clients are slow.
> 
> There was a couple of changes since v2, but all of them was 
> related to attributes delivered with listing.
> 
> Tigran.
> 
> P.S:
> 
>  > - contribute to NFSv4.2
> yes
>  >- implement an NFSv4.2 client and/or server at  >

---

From: sfaibish <[email-masked]>
Date: Fri, 28 Aug 2009 09:47:21 -0400

>> especially if clients are slow.
I think this will address the large number of objects scalability issue  
that I
proposed to address in 4.2. I am also supportive and interested to  
contribute
to any draft. I will also look how to address it in FedFS.

-- 
Best Regards

Sorin Faibish

         EMC²
where information lives

---

From: Benny Halevy <[email-masked]>
Date: Sun, 30 Aug 2009 11:21:37 +0300

> of a directory listing which valid until closedir is called. But there

http://www.opengroup.org/onlinepubs/009695399/functions/readdir.html:
...
If a file is removed from or added to the directory after the most recent call
to opendir() or rewinddir(), whether a subsequent call to readdir() returns an
entry for that file is unspecified.

This is not how I would define a snapshot... :-)

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Sun, 30 Aug 2009 11:33:28 -0400

> This is not how I would define a snapshot... :-)

Read the entries for telldir() and seekdir(). They're the functions that
really impose a "snapshot" behaviour...

Cheers
  Trond

---

From: Benny Halevy <[email-masked]>
Date: Sun, 30 Aug 2009 21:01:31 +0300

> really impose a "snapshot" behaviour...

Note the informative part of seekdir's description, in particular:
"The definition of seekdir() and telldir() does not specify whether,
when using these interfaces, a given directory entry will be seen at all,
or more than once."

The requirement that a location returned by telldir() will be valid
as input to seekdir() as long as rewinddir() was not called can
be satisfied with the READDIR cookie, but there's no guarantee that
after the directory is opened all the entries that were in the directory
will be returned by subsequent readdir()s and no new entries will
be seen.

That said, it's important that all entries that weren't deleted
after opendir() will be returned by subsequent readdir()s.  If that
can't be guaranteed then readdir() should fail ENOENT and the application
can then restart the directory scan using rewinddir()
(or closedir() and opendir()).  Let's just not call this snapshot
or "snapshot" behavior since this is too strong a definition.

Benny