Subject: [nfsv4] RFC 7530: Filehandle of opened file after the REMOVE

From: Marcel Telka <[email-masked]>
Date: Fri, 9 Dec 2016 23:25:00 +0100

Hi,

I'm not sure I understand RFC 7530 properly.  Here is the scenario I'm not sure
about:

1. clientA opens a file named with OPEN4_SHARE_DENY_NONE.
2. clientB opens the (same) file with OPEN4_SHARE_DENY_NONE.
3. clientB removes the file using the REMOVE operation.

In section 16.26.5. there is this:

   o  If the file was not opened with OPEN4_SHARE_DENY_WRITE or
      OPEN4_SHARE_DENY_BOTH, the server SHOULD delete the file's
      directory entry.  However, until the last CLOSE of the file, the
      server MAY continue to allow access to the file via its
      filehandle.

The above clearly says that for clientB the filehandle for the file after the
REMOVE operation MAY be valid.  The clientB just removed the file, so it should
not be surprised the filename (obtained in step 2) is no longer valid.

What is unclear to me is clientA.  Is it okay for the server to 'forget' the
filehandle even for clientA?  If the filehandle won't be valid for clientA
(after the REMOVE by clientB), then clientA might lost the access to the file
anytime, and it have no easy chance to prevent that (and no way to recover from
that).  This sounds unfortunate and it breaks the traditional unix semantics:
the file is there while I hold it opened.  The only way to make sure the file
is available until the final CLOSE seems to be to open the file with
OPEN4_SHARE_DENY_WRITE or OPEN4_SHARE_DENY_BOTH, which sounds like overkill.

Am I overlooking something?

Thank you.

-- 
+-------------------------------------------+
| Marcel Telka   e-mail:   [email-masked]  |
|                homepage: http://telka.sk/ |
|                jabber:   [email-masked] |
+-------------------------------------------+

---

From: David Noveck <[email-masked]>
Date: Sat, 10 Dec 2016 07:20:18 -0500

>The above clearly says that for clientB the filehandle for the file
> after the REMOVE operation MAY be valid.

As I read it, it MAY be either valid or invalid and this applies to
validity as perceived by any client.

> The clientB just removed the file, so it should not be surprised
> the filename (obtained in step 2) is no longer valid.

He shouldn't, but if he was a stickler for unix semantics, he would have a
problem.

> What is unclear to me is clientA.  Is it okay for the server to
> 'forget' the filehandle even for clientA?

It is allowed by the spec.

> If the filehandle won't be valid for clientA (after the REMOVE
> by clientB), then clientA might lost the access to the file anytime, >
and it have no easy chance to prevent that (and no way to recover > from
that).

> This sounds unfortunate and it breaks the traditional unix
> semantics: the file is there while I hold it opened.

True but people have been OK with this for two reasons:

   1. Most clients will implement NFSv3 sillyrename which is alluded to in
   the paragraph above the bulltets you quote.  Interestingly, it is a
   "should" rather than a "SHOULD", since interoperability is not affected.
   2. In the case of other clients, NFS users may have have been bringing
   expectations formed on the basis of experiences with NFSv3, and from an
   NFSv3 perspective, this server behavior is unexceptionable.

A possible extension to NFSv4.2 could provide an attribute to allow the
client to determine how the server dealt with this issue.

> The only way to make sure the file is available until the final
> CLOSE seems to be to open the file with
> OPEN4_SHARE_DENY_WRITE or
> OPEN4_SHARE_DENY_BOTH,

I'm don't think you can be sure of this, since

   1. In speaking of the directory entry it only says "SHOULD NOT"
   2. The text doesn't say explicitly that processing should stop at this
   point, with an error returned.
   3. there seems to be no text at all regarding continued access to the
   file by existing opens in the DENY case.  This may need some sort of
   correction but I'm not sure at this point exactly what sort of correction
   we could put in, given the need to be compatible with all existing
   implementations.

> which sounds like overkill.

Not exactly:

   1. It prevents stuff you don't want to prevent
   2. It is not guaranteed to prevent what you really want to prevent.

> Am I overlooking something?

I think you are overlooking sillyrename.

In your example either or both of A and B might by implementing
silllyrename and it isn't clear how things would work if both A and B did.
:-(

I think it would help if we considered this issue in historical context.
Making NFSv4 stateful was a wrenching change and it is not surprising if we
have adapted to that change incompletely.  Some of the IESG commentary
about RFC3530 expressed reserve or uncertainty (although no actual
objection) about making this change, saying essentially "Well if you guys
*really* want to do this, OK".

On Fri, Dec 9, 2016 at 5:25 PM, Marcel Telka <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Marcel Telka <[email-masked]>
Date: Tue, 13 Dec 2016 11:35:47 +0000

> He shouldn't, but if he was a stickler for unix semantics, he would have a

I think all unix NFS clients usually expect the unix semantics.
They definitely do so (at least) for their local files...

>    "should" rather than a "SHOULD", since interoperability is not affected.

The paragraph you are referring to is copied (almost verbatim)
from RFC 1813, section 3.3.12.  Unfortunately.  I would prefer to do not
have it here and instead require the server to allow access to the removed
file until the last close.  I'm late, I know.

>    NFSv3 perspective, this server behavior is unexceptionable.

Yes, but from the NFSv3 perspective the NFSv4 is not needed at all :-).

Maybe we should add a note that "if the server wants to support the unix
semantics (a removed file is accessible until the last close) then the
continued access to the file via its file handle until the last
CLOSE is essential".

> client to determine how the server dealt with this issue.

Either that, or just the note I suggested above.  Or even better: both.

Please note that without the continued access to the file via its file
handle until the last CLOSE there seems to be no way how to reliably
implement the unix semantics for NFS clients.  And no, the sillyrename
is not the answer, because if it is, then all clients would need to
create a silly name for every opened file to make sure no other client
(or even a local process at the server) will remove their file while
it is still open by them.  The other problem with this approach
would be: how to prevent other clients to remove my silly name?

>    1. In speaking of the directory entry it only says "SHOULD NOT"

True.  I misinterpreted SHOULD NOT as SHALL NOT.

> I think you are overlooking sillyrename.

Unfortunately, the sillyrename works only for the client that currently
have the file opened.  If the client is removing a file it does not have
opened, it does not do the sillyrename (at least the clients I'm
working with does not do so).

> silllyrename and it isn't clear how things would work if both A and B did.

In my example the clientA have no reason to implement the silllyrename
because it didn't removed the file.

Thank you.

---

From: David Noveck <[email-masked]>
Date: Tue, 13 Dec 2016 09:49:01 -0500

> I'm late, I know.

I guess I can't resist a (barely) relevant movie quote, from *The Big
Short:*

Dr. Michael Burry:  I may have been early, but I'm not wrong.

Martin Blaine: It's the same thing!

You're late with this but there would have been no way that you could have
been on time.
If you raised this issue in time for RFC3530, I don't think anyone would
have listened to
you.  You could have stuck to your position, made yourself very unpopular
and
prevented NFSv4 from happening.

RFC3530, RFC7530, RFC 5661 are immutable.  The only way they can be changed
is
via errata, and your judgment, that we made the wrong choice here is not a
good basis
for one.

I think this could be addressed in NFSv4 going forward via an extension.

---

From: Marcel Telka <[email-masked]>
Date: Tue, 13 Dec 2016 15:58:25 +0000

> RFC3530, RFC7530, RFC 5661 are immutable.  The only way they can be changed

RFC 3530 is maybe immutable, but also obsoleted, so nobody cares
about it much these days. :-)

Similarly as nobody cares about RFC 3010 now.

> via errata, and your judgment, that we made the wrong choice here is not a

No, I do not suggest that the choice made was wrong.  The current wording
allows to implement (maybe) simpler NFSv4 servers that do not need to bother
with preserving file handles after the REMOVE (if they do not worry about
unix).  This is good.

> I think this could be addressed in NFSv4 going forward via an extension.

Why not to use the note I suggested (properly rephrased, of course) even
for RFC 7530 (as errata)?

This one:

"If the server wants to support the unix semantics (a removed file is
accessible until the last close) then the continued access to the file
via its file handle until the last CLOSE is essential."

It won't change this (or anything else in RFC 7530, AFAIK):

It will just add a note (or recommendation, suggestion, clarification,
whatever) for implementors who want to make sure their NFSv4 servers
are properly prepared to support the unix semantics for NFSv4 clients.

And, ... what implementors do now with RFC 7530?  If they want to support
the unix semantics they need to follow the note above albeit it is not
written in the RFC.

Thanks.

---

From: David Noveck <[email-masked]>
Date: Tue, 13 Dec 2016 11:20:41 -0500

> the unix semantics they need to follow the note above albeit it is not

Some implementors already do this, and your note won't affect them.

The problem is with those that don't.  Why would your note cause them to
change course?  I think they will basically figure that clients are going
to do
a silly rename anyway, so they don't have to bother changing their code.

The problem is that NFSv3 made this a client responsibility and NFSv4,0 left
things as they were.  This is not just an issue for Unix.  Making sure an
open file
is not removed out from under the client is a good thing for other OS's as
well.

---

From: Marcel Telka <[email-masked]>
Date: Tue, 13 Dec 2016 16:56:39 +0000

> change course?  I think they will basically figure that clients are going

My note will suggest them that if they want the unix semantics they will
be clearly instructed what they need to implement.

Now, without the note, it is not clear what exactly needs to be done to get
the unix semantics.

> a silly rename anyway, so they don't have to bother changing their code.

Sure, they have an option to do not implement the unix semantics, if they will
decide so.

Anyway, it is not possible to implement unix semantics using the silly
rename only.  Or, do you suggest that all removes should be done as
silly renames?

> is not removed out from under the client is a good thing for other OS's as

How can a NFSv4 client do that without help from either server or other
client(s)?

Please return back to my example in the first mail.  What needs the clientA
to do to make sure its file does not disappear?  If the answer is "nothing, it
is responsibility of clientB to do the silly rename" then:

1) we do not need the REMOVE operation :-)
2) how we will make sure that _all_ clients will do the silly
    rename (including NFSv2 clients, NFSv3 clients, NFSv4.1 clients,
    NFSv4.2 clients, NFSv4.3 clients, NFSv5.0 clients, CIFS
    clients, and local processes)?

Thanks.

---

From: Marcel Telka <[email-masked]>
Date: Tue, 13 Dec 2016 17:19:02 +0000

Citát Trond Myklebust <[email-masked]>:

> On Tue, Dec 13, 2016 at 11:56 AM, Marcel Telka <[email-masked]> wrote:
>
> That's hilarious...

Please elaborate.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 13 Dec 2016 12:13:26 -0500

That's hilarious...

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 13 Dec 2016 12:43:36 -0500

>>> be clearly instructed what they need to implement.
RFC7530 was never intended to serve as the basis for a new standard; it was
an update of RFC3530 and was proposed solely in order to make the
description of the NFSv4 spec comply with the existing implementations. The
main requirement was that we must not change the wire protocol unless there
was mismatch between the spec and the implementations.

Now you're asking for a change to a 13 year old protocol for which there
has not been a new implementation in almost a decade. Are you writing an
implementation of this change?

---

From: Marcel Telka <[email-masked]>
Date: Tue, 13 Dec 2016 18:17:34 +0000

> implementation of this change?

Sorry, but I probably didn't worded my mails properly.  And, unfortunately,
I do not know what should I reword to make it more clear.  Please let me know
what in my mails suggests you that I propose the wire protocol change.  
  Thanks.

In any case: I'm not asking for a protocol change (or at least it is not
my intention). I'm asking for a note that won't change over the wire protocol
at all.  It would just document the common practice used (and implemented)
by many current NFSv4.0 servers (for example Linux NFS server).

Yes, we recently found that illumos NFSv4.0 server does not follow this
(maybe Solaris is same, I didn't tested) so the unix semantics is not  
supported
there.  And yes, there is an attempt to implement it.

And, please note that the implementation in Linux NFS server is not denied by
the RFC 7530.  The suggested note won't change that.  The note just says:

If you want a feature A, then you should implement B.  Where B is optional.

A == unix semantics on NFS clients
B == access to removed files via their file handle until the last CLOSE

Currently, the RFC 7530 says:

If you want, you can implement B.  Where B is optional.

Thank you.

---

From: David Noveck <[email-masked]>
Date: Wed, 14 Dec 2016 05:56:02 -0500

It appears that you want an informational document saying, more or less:

   - If the server does not want clients to be discomfited by open files
   being removed, since such behavior is disallowed by typical OS (e.g.Unix)
   semantics, the server can avoid this situation by delaying the actual
   removal of the file until last close, as allowed by RFC7530.
   - The use of rename by clients as a substitute for remove, normally
   known as "silly rename", has significant problems, since removes can happen
   on nodes that do not have the file open.

If this is what you want, then you can write an I-D and submit it.

---

From: Marcel Telka <[email-masked]>
Date: Wed, 14 Dec 2016 11:21:12 +0000

> If this is what you want, then you can write an I-D and submit it.

Yes, this is exactly what I want to see.

Thank you.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 14 Dec 2016 09:28:41 -0500

> Yes, this is exactly what I want to see.
There are 4 main problems that are inadequately discussed in any of the
existing RFCs, and that you need to address before we can consider
replacing sillyrename (which is well established today, and well understood
by most users).

1) How does the client identify that the server supports this functionality?
2) What functionality is needed on the underlying filesystems on the server?
3) How does the server function in the case of a reboot? What can the
client expect in terms of recoverability?
4) How does the client in practice perform recovery?
   a) In the case of server reboots.
   b) In the case of lease timeouts/network partitions

Note that the lack of open-by-filehandle in NFSv4.0 makes 4.b) more
difficult than it should otherwise be.

---

From: [email-masked] (J. Bruce Fields)
Date: Tue, 27 Dec 2016 09:44:14 -0500

> 1) How does the client identify that the server supports this functionality?

I think this is what OPEN4_RESULT_PRESERVE_UNLINKED in 5661 was meant to
do.  It'd be interesting to try an implementation and see if your other
points are addressed, but I haven't thought about it in a long time.

--b.

---

From: David Noveck <[email-masked]>
Date: Tue, 27 Dec 2016 13:13:44 -0500

Bruce wrote:

> do.

In fact it was.  See the sixth bullet in section 1.8 of RFC5661.

The problem I see is that it doesn't solve the problem that Marcel has
pointed out.  If client A
opens the file and client B removes it, there is no way it can decide
whether or not to do a
silly rename.  Since he hasn't opened, the file he doesn't see the flag
indicating it is not
needed.  Also, since the file is not open he has no indication that it is
needed either.  So client
B will do a REMOVE in all cases.

Server implementations may or may not preserve the file until last close.
For those that do,
everything will work out OK, while for those that don't, there isn't much
that the client can
do.  If he did find out the server did not have the support, he has o way
to defer the remove
until last close because he has no way of finding out when this happens,

> There are 4 main problems that are inadequately discussed in any of the
existing RFCs,
> and that you need to address before we can consider replacing sillyrename

It appears it has already been considered and spec'd but the spec may not be
adequate.  Sigh!

> (which is well established today, and well understood by most users).

Most people understand why it works when it does but don't understand why
it doesn't
work in some cases.  So they are happy until they aren't.

> 1) How does the client identify that the server supports this
functionality?

The problem is that the client who doesn't open file, the troublesome case,
is exactly the one in which
he receives no information.  The client who does the open finds out silly
rename is not needed, but, in
this case, silly rename seems to work OK.

> 2) What functionality is needed on the underlying filesystems on the
server?

Almost all filesystems have the ability to defer deletion until the last
close.  If they don't, they
are not very usable.  Servers which don't support this are not suffering
from a lack of filesystem
functionality.  Instead, the problem seems to be that some servers have an
NFSv4 open which
either doesn't connect the to the underlying open functionality or there is
no underlying open functionality.

This is addressed b section 18.16.3 of rfc5661.  The third bullet on page
 449 says:

Furthermore, the server promises to preserve the file through the
grace period after server restart, thereby giving the client the
opportunity to reclaim its open.

>   a) In the case of server reboots.

I think he just recovers this opens within the grace period.

>   b) In the case of lease timeouts/network partitions

This case is not addressed RFC5661, so far as I can see.  The typical
handling, in which
the locks are all dropped would cause the file to go away.  Courtesy locks
could address the
problem, but the spec doesn't that have to be implemented.

True. In v4.1, a client can try an open-by-filehandle and take advantage of
any courtesy locks that the server
has retained.

On Tue, Dec 27, 2016 at 9:44 AM, J. Bruce Fields <[email-masked]>
wrote:

> > >>    known as "silly rename", has significant problems, since removes
> can
> > >> happen

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 27 Dec 2016 21:24:36 +0100

> On Dec 27, 2016, at 19:13, David Noveck <[email-masked]> wrote:
> 
> Bruce wrote:
> 
> The problem I see is that it doesn't solve the problem that Marcel has pointed out.  If client A
> opens the file and client B removes it, there is no way it can decide whether or not to do a 
> silly rename.  Since he hasn't opened, the file he doesn't see the flag indicating it is not 
> needed.  Also, since the file is not open he has no indication that it is needed either.  So client

As far as I can tell, Marcel is mainly interested in NFSv4.0. RFC5661 does not address any of the cases he was describing.

> 
> Server implementations may or may not preserve the file until last close.  For those that do, 
> everything will work out OK, while for those that don't, there isn't much that the client can 
> do.  If he did find out the server did not have the support, he has o way to defer the remove 
> > There are 4 main problems that are inadequately discussed in any of the existing RFCs, 
> Most people understand why it works when it does but don't understand why it doesn't
> work in some cases.  So they are happy until they aren’t.

Umm…. If the solution works with unlink-on-close then you all do realize that it MUST always also work with sillyrename, right? Having a third party client remove a file named ‘.nfsXXXXX’ is just a special case of having it remove a file with a generic name.

IOW: I could argue that the problem here can be considered to be purely a server problem and irrelevant to the actual delete strategy chosen by the client.

> The problem is that the client who doesn't open file, the troublesome case, is exactly the one in which
> he receives no information.  The client who does the open finds out silly rename is not needed, but, in
> Almost all filesystems have the ability to defer deletion until the last close.  If they don't, they
> are not very usable.  Servers which don't support this are not suffering from a lack of filesystem
> functionality.  Instead, the problem seems to be that some servers have an NFSv4 open which 
> either doesn't connect the to the underlying open functionality or there is no underlying open functionality.

Sure, but in order to be reboot safe, you need to go well beyond the functionality that you describe. As far as I can tell, you need to defer the garbage collection that would normally occur when the server comes back up again until after the state reclaim has occurred (as you describe below).

> 
> > 3) How does the server function in the case of a reboot? What can the client expect in terms of recoverability?
> 
> This is addressed b section 18.16.3 of rfc5661.  The third bullet on page  449 says:
> 
> Furthermore, the server promises to preserve the file through the grace period after server restart, thereby giving the client the opportunity to reclaim its open.

Again, this is specified for NFSv4.x (x>0) but not for x=0. It is not clear to me that older NFSv4 servers will have any of this functionality.

> This case is not addressed RFC5661, so far as I can see.  The typical handling, in which
> the locks are all dropped would cause the file to go away.  Courtesy locks could address the
> > Note that the lack of open-by-filehandle in NFSv4.0 makes 4.b) more difficult than it should otherwise be.
> 
> True. In v4.1, a client can try an open-by-filehandle and take advantage of any courtesy locks that the server 
> has retained.
> 
> On Tue, Dec 27, 2016 at 9:44 AM, J. Bruce Fields <[email-masked] <mailto:[email-masked]>> wrote:
> > On Wed, Dec 14, 2016 at 6:21 AM, Marcel Telka <[email-masked] <mailto:[email-masked]>> wrote:
> >
> > > Citát David Noveck <[email-masked] <mailto:[email-masked]>>:
> > >
> > [email-masked] <mailto:[email-masked]>
> > https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>
>

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 27 Dec 2016 22:00:18 +0100

> On Dec 27, 2016, at 21:24, Trond Myklebust <[email-masked]> wrote:
> 
>> 
>> On Dec 27, 2016, at 19:13, David Noveck <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Bruce wrote:
>> 

Correction: most of the problem is orthogonal to the client delete strategy. The one place where the client can put its foot in its mouth is ‘rm -rf’, where the sillyrename does prevent the removal of the directory from succeeding.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 28 Dec 2016 08:08:06 +0200

On Dec 27, 2016 11:00 PM, "Trond Myklebust" <[email-masked]> wrote:

As far as I can tell, Marcel is mainly interested in NFSv4.0. RFC5661 does
not address any of the cases he was describing.

existing RFCs,

Umm…. If the solution works with unlink-on-close then you all do realize
that it MUST always also work with sillyrename, right? Having a third party
client remove a file named ‘.nfsXXXXX’ is just a special case of having it
remove a file with a generic name.

IOW: I could argue that the problem here can be considered to be purely a
server problem and irrelevant to the actual delete strategy chosen by the
client.

Correction: most of the problem is orthogonal to the client delete
strategy. The one place where the client can put its foot in its mouth is
‘rm -rf’, where the sillyrename does prevent the removal of the directory
from succeeding.

A different silly rename strategy could be to a root-based directory, e.g.
/trashcan. That would prevent the rm -rf issue, no?

Benny

Sure, but in order to be reboot safe, you need to go well beyond the
functionality that you describe. As far as I can tell, you need to defer
the garbage collection that would normally occur when the server comes back
up again until after the state reclaim has occurred (as you describe below).

Again, this is specified for NFSv4.x (x>0) but not for x=0. It is not clear
to me that older NFSv4 servers will have any of this functionality.

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 28 Dec 2016 13:59:34 +0100

> On Dec 28, 2016, at 07:08, Benny Halevy <[email-masked]> wrote:
> 
> 
> 
> On Dec 27, 2016 11:00 PM, "Trond Myklebust" <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> On Dec 27, 2016, at 21:24, Trond Myklebust <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>>> 
> A different silly rename strategy could be to a root-based directory, e.g. /trashcan. That would prevent the rm -rf issue, no?

You need a guarantee that the user holds the same file removal permissions in that directory; that’s something the client cannot enforce.

---

From: Bruce James Fields <[email-masked]>
Date: Wed, 28 Dec 2016 21:47:03 -0500

On Tue, Dec 27, 2016 at 09:24:36PM +0100, Trond Myklebust wrote:
> functionality that you describe. As far as I can tell, you need to
> defer the garbage collection that would normally occur when the server
> comes back up again until after the state reclaim has occurred (as you
> describe below).

I never seriously worked on it, but for a while I was in the habit of
running it by people.  Christoph Hellwig thought it was doable (I think
he suggested some sort of callback from the filesystem during the
garbage collection, possibly because he had in mind some other
application for that--but my memory may be wrong).  Chris Mason didn't
like the idea at all.  He asked what we expect to happen on fsck, or if
the filesystem gets mounted without nfs getting started, or... some
other scenarios I forget.

We could do the same silly rename tricks on the server side.  Something
like: create a directory with an unlikely name in the root of the
export, rename files there on REMOVE.  Possible problems:

	- you'll never be able to completely hide that directory.  But
	  maybe we could get some sort of filesystem support for a
	  hidden directory.
	- what if the remove is from a non-nfs user?  Maybe we could get
	  some kind of notification on unlink of an open file, maybe we
	  just live with continuing not to handle this particular case.
	- I guess we need to tell nfs users that the inode's link count
	  is zero?  (Any local users will still see it as one, though?)

--b.

---

From: Christoph Hellwig <[email-masked]>
Date: Thu, 29 Dec 2016 08:48:30 +0100

> other scenarios I forget.

The way open but unlinked files are handled by modern transaction
file systems is that the file system has a list of those inodes
(in XFS this is the unlinked inode list in the allocation group header,
other file systems use different terminologies and slightly different
technics, e.g. in ext4 the list is global for the whole file system).

After an unclean shutdown when file system recovery is run we'll perform
the deferred delete for all the inodes on the unlinked inode list.
At that point the file system could in theory inform NFSD about that
fact.  But at least as far as the current Linux kernel is concerned (
sorry for delving into implementation details, but I guess this is still
easier to understand than an abstract discussion) at the point where
file system performs recovery NFSD has not been started, or at least doesn't
know about the file system yet.   We could still persist that information
somewhere, or use a flag to delay the deletion of unlinked inodes until
NFSD runs.

> export, rename files there on REMOVE.  Possible problems:

Personally I'd love to see sillyrename die.  It's a major pain for
getting sensible semantics out of NFS.

> 	  maybe we could get some sort of filesystem support for a

The unlinked inode list is almost a directory, except that it doesn't
have names for the entries, you can only find inodes on it by the inode
number and generation (aka NFS file handle).

---

From: David Noveck <[email-masked]>
Date: Thu, 29 Dec 2016 05:53:36 -0500

> You need a guarantee that the user holds the same file removal
permissions in that directory; that’s something the client cannot enforce.

The client needs that to do the eventual remove at last close.  To do the
rename you also need ADD_FILE permissions on the target directory.

Within the NFSV4 ACL model, you can have a directory for which a user has
DELETE_CHILD permissions without having ADD_FILE, so that the silly rename
(within the same directory) would fail with a permission error.  Not sure
what existing clients do about this.  My best guess is that the user
process gets a permission error on his remove, despite having DELETE_CHILD
permissions on the directory.  Such directories are unlikely to exist with
current clients and servers but the NFSv4 RFCs allow them to be created.

On Wed, Dec 28, 2016 at 7:59 AM, Trond Myklebust <[email-masked]> wrote:

> You need a guarantee that the user holds the same file removal permissions
> in that directory; that’s something the client cannot enforce.
>
>
> Benny
>
>
>
> Again, this is specified for NFSv4.x (x>0) but not for x=0. It is not
> clear to me that older NFSv4 servers will have any of this functionality.
>
>