Subject: [nfsv4] ESXi NFSv4.1 client's use of ReclaimComplete

From: Rick Macklem <[email-masked]>
Date: Mon, 9 Jul 2018 22:40:21 +0000

Hi,

Here's another one from ESXi client testing...
Although FreeBSD is not a supported server for the ESXi NFSv4.1 client,
someone has been doing testing of the FreeBSD NFSv4.1 server for it and packet
traces have shown the following:
- The client does a ReclaimComplete with rca_one_fs == TRUE after acquiring a
  new clientid via ExchangeID and CreateSession.
  The FreeBSD server replies NFS4ERR_NOTSUPP.

RFC5661 says:
   Whenever a client establishes a new client ID and before it does the
   first non-reclaim operation that obtains a lock, it MUST send a
   RECLAIM_COMPLETE with rca_one_fs set to FALSE, even if there are no
   locks to reclaim.  If non-reclaim locking operations are done before
   the RECLAIM_COMPLETE, an NFS4ERR_GRACE error will be returned.
This seems clear to me and the FreeBSD server is expecting a ReclaimComplete
with rca_one_fs == FALSE.

Also, the description at the beginning of the section says:
   A RECLAIM_COMPLETE operation is used to indicate that the client has
   reclaimed all of the locking state that it will recover, when it is
   recovering state due to either a server restart or the transfer of a
   file system to another server.  There are two types of
   RECLAIM_COMPLETE operations:

   o  When rca_one_fs is FALSE, a global RECLAIM_COMPLETE is being done.
      This indicates that recovery of all locks that the client held on
      the previous server instance have been completed.

   o  When rca_one_fs is TRUE, a file system-specific RECLAIM_COMPLETE
      is being done.  This indicates that recovery of locks for a single
      fs (the one designated by the current filehandle) due to a file
      system transition have been completed.  Presence of a current
      filehandle is only required when rca_one_fs is set to TRUE.
I think that "file system transition" here was meant to refer to "transfer of
a file system to another server" above, but I can see room for confusion here.

Then the para. after the first one above says:
   Similarly, when the client accesses a file system on a new server,
   before it sends the first non-reclaim operation that obtains a lock
   on this new server, it MUST send a RECLAIM_COMPLETE with rca_one_fs
   set to TRUE and current filehandle within that file system, even if
   there are no locks to reclaim.  If non-reclaim locking operations are
   done on that file system before the RECLAIM_COMPLETE, an
   NFS4ERR_GRACE error will be returned.
I believe "file system on a new server" refers to "after a file system has
been transferred to a new server", but I can see how this might be interpreted
differently. Maybe "new server" meaning one that is just being mounted.

Since FreeBSD never transfers a file system to a different server, it
returns NFS4ERR_NOTSUPP for the "rca_one_fs == TRUE" case.

Have I misinterpreted the above?

If not, I am surprised that this client works for any NFSv4.1 server, but
apparently it does.

rick
ps: I have an ugly work around, that basically assumes rca_one_fs is always
    FALSE and returns NFS_OK even if Reclaim_Complete has already been done.
    However, I don't really like doing this. Anyone have a better suggestion?

---

From: David Noveck <[email-masked]>
Date: Mon, 9 Jul 2018 19:53:25 -0400

Thanks for pointing this issue out.

> The client does a ReclaimComplete with rca_one_fs == TRUE after acquiring
a
>  new clientid via ExchangeID and CreateSession.

What is  the current file handle when it does this?

As you point out, some of the text in RFC5661, regarding the use of one_fs
could be clearer.
Unfortunately, It may be hard to clarify this text given what servers may
have done to
adapt to this client's behavior.   I think that, once mv1-msns-update is
published, we
are going to have servers that do implement fs transfer and, in that case,
the current
confusion about one_fs will have to be addressed, without breaking any
existing
implementations.

Despite the lack of clarity you mention the following is reasonably clear:

>   locks to reclaim.

So this client is not doing what the spec says it MUST do.    Apparently,
servers
have adapted to this behavior, perhaps in the way that you  propose at the
end
of your mail and perhaps in other ways.

> If non-reclaim locking operations are done before

In this case, it seems clear thar "the RECLAIM_COMPLETE" refers to one done
with one_fs set to FALSE and so implies that the server will be returning
NFS4ERR_GRACE, which I don't think is happening.

This is a situation that can be lived with, but when there are servers that
do
implement an fs-specific grace period, it will be necessary to distinguish
RECLAIM_COMPLETE's done with one_fs set to TRUE because it is really
intended to be part of resolving fs transfer and those that are not done for
that prpose and where the server is simply hacking around the behavior you
noted.  Sigh!

On Mon, Jul 9, 2018 at 6:40 PM, Rick Macklem <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Rick Macklem <[email-masked]>
Date: Tue, 10 Jul 2018 20:01:24 +0000

David Noveck wrote:
>> The client does a ReclaimComplete with rca_one_fs == TRUE after acquiring a
It is for a file called "install", but because the testing was done by others and I
don't know what their exported file system hierarchy is, I don't know if it was
the file system they mounted or some other file system that forms a subtree of
it.

>As you point out, some of the text in RFC5661, regarding the use of one_fs could be >clearer.
>Unfortunately, It may be hard to clarify this text given what servers may have done to
>adapt to this client's behavior.   I think that, once mv1-msns-update is published, we
>are going to have servers that do implement fs transfer and, in that case, the current
>confusion about one_fs will have to be addressed, without breaking any existing
>implementations.
>
>So this client is not doing what the spec says it MUST do.    Apparently, servers
>have adapted to this behavior, perhaps in the way that you  propose at the end
For the FreeBSD server, they have been testing with a "hack" that does what I
described in the "ps:", so the "rca_one_fs == FALSE" was faked.
Without the hack, the mount was not usable from what I understand.

>This is a situation that can be lived with, but when there are servers that do
I don't have any plans w.r.t. implementing file system transfer at this time, but
others who are might want to do testing with a VMware client to see what happens.
(I'll admit I'm really tempted to implement the hacks based on the VMware
 client's eia_client_impl_id, but I know I shouldn't do it.;-)

Thanks for the comments, rick
[stuff snipped]

---

From: David Noveck <[email-masked]>
Date: Wed, 11 Jul 2018 06:36:11 -0400

Regarding use of eia_client_impl_id, actually RFC5661 says "MUST NOT".

I've been considering ways to allow servers that support fs migration to do
so while still supporting behavior such as that of the troublesome VMware
client.  I intend to discuss the issue in migration-issues-16 and there
will be an opportunity to discuss it (remotely if that is most convenient)
at IETF102. My hope is to arrive at a suitable treatment to include in
mv1-msns-update-02.

On Jul 10, 2018 4:01 PM, "Rick Macklem" <[email-masked]> wrote:

>What is  the current file handle when it does this?
It is for a file called "install", but because the testing was done by
others and I
don't know what their exported file system hierarchy is, I don't know if it
was
the file system they mounted or some other file system that forms a subtree
of
it.

>As you point out, some of the text in RFC5661, regarding the use of one_fs
could be >clearer.
have done to
published, we
the current
existing
servers
end
>NFS4ERR_GRACE, which I don't think is happening.
For the FreeBSD server, they have been testing with a "hack" that does what
I
described in the "ps:", so the "rca_one_fs == FALSE" was faked.
Without the hack, the mount was not usable from what I understand.

>that prpose and where the server is simply hacking around the behavior you
I don't have any plans w.r.t. implementing file system transfer at this
time, but
others who are might want to do testing with a VMware client to see what
happens.
(I'll admit I'm really tempted to implement the hacks based on the VMware
 client's eia_client_impl_id, but I know I shouldn't do it.;-)

---

From: Rick Macklem <[email-masked]>
Date: Wed, 11 Jul 2018 12:06:02 +0000

>Regarding use of eia_client_impl_id, actually RFC5661 says "MUST NOT".
Yes, I am aware of this. I think that, since the FreeBSD server doesn't do file
system transfers, it won't see a "rca_one_fs == TRUE" from other clients
and "pretending it was FALSE" can be done for all clients pretty safely.
It will not be enabled by default and will have to be configured, but shouldn't
need use of eia_client_impl_id.

There are a couple of really ugly ones in the packet traces for ESXi6.5 when the server
reboots, but I don't think the server can safely work around those. (I recently
heard from the main tester that ESXi6.7 works better, but haven't seen packet
traces for it yet.)
These I am pretty sure are client bugs (both seen after a server reboot):
- eia_clientowner.co_verifier flip flops from another hex value to all 0s and back
   during multiple EXCHANGE_ID operations. I think the RFC is clear w.r.t. that a
   change in the co_verifier indicates a client restart, which hasn't happened.
   - The only hack I came up with, which did make the tester happy, was to ignore the
     transition and not delete Open/Lock state on the server, but I think that is too ugly
     to put in a released system.
- The delegate_type argument for Open/ClaimPrevious is just random bits and
  not one of the enumerated types. (Since it also sets OPEN_SHARE_ACCESS_WANT_NO_DELEG in the arguments, it can be worked
around by assuming OPEN_DELEGATE_NONE, which seems to make the client happy.)
>From the above two cases, I think you can see why I am tempted (but I am
resisting the temptation) to use eia_client_impl_id.;-)

>I've been considering ways to allow servers that support fs migration to do so while >still supporting behavior such as that of the troublesome VMware client.  I intend to >discuss the issue in migration-issues-16 and there will be an opportunity to discuss it >(remotely if that is most convenient) at IETF102. My hope is to arrive at a suitable >treatment to include in mv1-msns-update-02.
Btw, someone gave VMware a bug report on this. I think it is in the "might be
fixed for a future release" category.
As such,  there is at least a chance that they will change to using rca_one_fs == FALSE at some point.

rick
[good stuff snipped]

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 11 Jul 2018 09:42:40 -0400

Is the client perhaps setting EXCHGID4_FLAG_UPD_CONFIRMED_REC_A in
these exchange id calls? If so, then that should not result in the
client losing state (See RFC5661, Section 18.35.4).

---

From: Rick Macklem <[email-masked]>
Date: Wed, 11 Jul 2018 21:06:30 +0000

>client losing state (See RFC5661, Section 18.35.4).
Not for the ESXi 6.5 packet traces I saw. It generates a whole bunch of
ExchangeID and CreateSession RPCs, such that the csa_sequenceid is almost
never the same as the eir_sequenceid of the last ExchangeID.
Sometimes, eventually it sends a CreateSession with the correct csa_sequenceid
and gets past this.
Then sometimes it works and sometimes it sends another ExchangeID, which
gets a reply with EXCHGID4_FLAG_CONFIRMED_R set in it and then just loops
resending the ExchangeID forever.
(I'll admit it found a resource problem in my server that I had to fix so it wouldn't
 kill the server when it did this.)
Then, if it does get past this, it throws Open/ClaimPrevious ops with random
garbage in the delegate_type argument and then, if you let it get past this
(the FreeBSD server replied NFS4ERR_BADXDR until I hacked it around that for him),
it gets upset when it sees NFS4ERR_GRACE replies to Read/Writes that it does
right after the Open/ClaimPrevious's.

As I mentioned, the tester has said ESXi 6.7 works better, but I don't know what,
if any, of these issues still exist until I see his packet traces.
(He also uses trunking, which I think makes the above worse than when there is
 a single TCP connection.)

It's a fun client to test a server with, rick
[again, good stuff snipped]

---

From: Rick Macklem <[email-masked]>
Date: Sun, 15 Jul 2018 20:10:00 +0000

Just as a follow up to this, the packet traces I've seen to ESXi 6.7
show it behaving much better than ESXi6.5 did.

It still seems to generate a ReclaimComplete with rca_one_fs == TRUE when
it first does a mount, but it also generates a ReclaimComplete with
rca_one_fs == FALSE at about the same time. (Usually one right after the
other, although I'm not yet sure if the ordering is fixed.)

All the other serious problems I was seeing aren't in these packet
traces. (I will wait and see if they show up in future packet traces,
but it certainly is a lot better, if not completely fixed for these other
issues.)

rick

________________________________________
From: nfsv4 <[email-masked]> on behalf of Rick Macklem <[email-masked]>
Sent: Wednesday, July 11, 2018 5:06:30 PM
To: Trond Myklebust
Cc: [email-masked]
Subject: Re: [nfsv4] ESXi NFSv4.1 client's use of ReclaimComplete

---

From: Rick Macklem <[email-masked]>
Date: Sun, 15 Jul 2018 23:50:08 +0000

>other, although I'm not yet sure if the ordering is fixed.)
Just fyi, I just looked at the Linux 4.17-rc2 kernel and the Linux server just returns
NFS_OK for the rca_one_fs == TRUE case.

I plan to do the same for the FreeBSD server (which currently returns an error
for the rca_one_fs case).

rick

---

From: David Noveck <[email-masked]>
Date: Mon, 16 Jul 2018 05:24:29 -0400

> NFS_OK for the rca_one_fs == TRUE case.

By  "*just* returns NFS_OK for the rca_one_fs == TRUE case" do you mean:

a) returns NFS_OK and does nothing else, i.e. essentially ignore the reqest.

WhilIe I think that is the right thing to do if the fs is one which is not
undergoing

mifgration and thus has no per-fs grace period, it would cause ESXi6.5 to

get NFS4ERR_GRACE when it tried to open a file.

b) returns NFS_OK after treating it as if rca_one_fs were FALSE.

I can see doing this since it allows ESXi6.5 to work, although this was not

intended by the spec.   This seems a case where "SHOULD NOT" is appropriate.

The problem with this is thet one of the RECLAIM_COMPLETEs done by ESXi6.7

would get NFS4ERR_COMPLETE_ALREADY.

c) Do somethng else (which I'm not going to speculate about now) that
allows both
ESXi6.5 and ESXi6.7 to work.

I'm worried about somehow accommodating c) and still allowing

RECLAIM_COMPLETE with rca_one_fs set to TRUE to work for its i
ntended purpose.

> for the rca_one_fs case).

Do you plan to do a, b), or c)?

---

From: Rick Macklem <[email-masked]>
Date: Mon, 16 Jul 2018 12:28:33 +0000

David Noveck wrote:
>By  "just returns NFS_OK for the rca_one_fs == TRUE case" do you mean:
>
>WhilIe I think that is the right thing to do if the fs is one which is not undergoing
Well, here's the code snippet. (It does check for a valid current FH, but that seemed
irrelevant to the discussion.)
nfsd4_reclaim_complete(struct svc_rqst *rqstp,
		struct nfsd4_compound_state *cstate, union nfsd4_op_u *u)
{
	struct nfsd4_reclaim_complete *rc = &u->reclaim_complete;
	__be32 status = 0;

	if (rc->rca_one_fs) {
		if (!cstate->current_fh.fh_dentry)
			return nfserr_nofilehandle;
		/*
		 * We don't take advantage of the rca_one_fs case.
		 * That's OK, it's optional, we can safely ignore it.
		 */
		return nfs_ok;
	}
As you can see, it is "a)".

>would get NFS4ERR_COMPLETE_ALREADY.
Yes. The "hacked patch" the testers were using also returned NFS_OK instead of
NFS4ERR_COMPLETE_ALREADY. (I do not plan on committing this patch to FreeBSD.)

ESXi6.5 didn't always do a ReclaimComplete with rca_one_fs == FALSE, but ESXi6.7
does seem to always do a ReclaimComplete with rca_one_fs == FALSE.

>c) Do somethng else (which I'm not going to speculate about now) that allows both
I now plan to do "a)" and just not try and support ESXi6.5. However, if the FreeBSD
"collective" has a strong opinion that ESXi6.5 should be supported, I can't be sure...

If I had known sooner that ESXi6.7 would have a lot of things "fixed" I wouldn't
have even started this discussion.

rick

On Sun, Jul 15, 2018 at 7:50 PM, Rick Macklem <[email-masked]<mailto:[email-masked]>> wrote:

rick

---

From: David Noveck <[email-masked]>
Date: Tue, 17 Jul 2018 06:25:20 -0400

> "collective" has a strong opinion that ESXi6.5 should be supported, I
can't be sure...

Thinking about supporting both at the same time, I've come to the conclusion
that the best ways to do this is to return NFS4_OK for per-fs
RECLAIM_COMPLETE
but note somewhere that theclient has done this, rather treat it like a
global
RECLAIM_COMPLETE.  Then, when the first non-reclaim locking request is
received you can check whether either a global or per-fs RECLAIM_COMPLETE
has
been done.   Although ugly, that's pretty safe if your are sure you will
never be the
desination server for fs migration.

> have even started this discussion.

In any case, it has been started and is just about complete now.  There
will be an
mv1-msns-update-02 that clarifies the handlling of RECLAIM_COMPLETE.
Thanks
for bringing these issues to my attention.