Subject: [nfsv4] What is a reasonable minimum lease time?

From: "Frank Filz" <[email-masked]>
Date: Wed, 12 Dec 2018 14:22:40 -0800

We have an issue with the Ganesha server with very short (2 second) lease
times. Ganesha uses a 1 second granularity for lease management, and
considers a time since last renewed equal to the lease time as too long. The
result is that the lease period may be short close to 2 seconds depending on
when within a given second things actually happened (so a last renew at 0.99
with a subsequent renew at 2.01 which is just over one second looks like 2
seconds to Ganesha and thus is >= the 2 second lease time and not good
enough. A simple change would be to change the >= to a >, which gives one
more second, but it still could result in the lease time being almost 1
second too short which is significant with a 2 second lease time. But if the
minimum reasonable lease time is more like 5 or 10 seconds, that 1 second
becomes less significant.

The bigger fix would be to use a finer grained time, but that adds
complexity, but if people really want to run with 2 second lease times and
it makes any kind of sense, we would need to make that change.

Thanks

Frank

---

From: Tom Talpey <[email-masked]>
Date: Thu, 13 Dec 2018 08:30:52 -0500

> it makes any kind of sense, we would need to make that change.

Actually I thnk your lease time is one second, based on the
description above. One second (or even two) seems ridiculously
short, to me.

The lease time needs to be at least as large as the time needed
to connect, establish a session (including authentication), and
for the client to complete full recovery of all state. And remember
if one client is doing this, it's likely many others are too.

The spec recommends 90. Why does your server choose a value two
orders of magnitude lower?

Tom.

---

From: "Frank Filz" <[email-masked]>
Date: Thu, 13 Dec 2018 05:49:25 -0800

> > We have an issue with the Ganesha server with very short (2 second)
> > lease times. Ganesha uses a 1 second granularity for lease management,
> > and considers a time since last renewed equal to the lease time as too
> > long. The result is that the lease period may be short close to 2
> > seconds depending on when within a given second things actually
> > happened (so a last renew at 0.99 with a subsequent renew at 2.01
> > which is just over one second looks like 2 seconds to Ganesha and thus
> > is >= the 2 second lease time and not good enough. A simple change
> > would be to change the >= to a >, which gives one more second, but it
> > still could result in the lease time being almost 1 second too short
> > which is significant with a 2 second lease time. But if the minimum
> > reasonable lease time is more like 5 or 10 seconds, that 1 second
becomes less
> significant.
> > complexity, but if people really want to run with 2 second lease times
> > and it makes any kind of sense, we would need to make that change.
> 
> Actually I thnk your lease time is one second, based on the description
above.
> One second (or even two) seems ridiculously short, to me.
> 
> The lease time needs to be at least as large as the time needed to
connect,
> establish a session (including authentication), and for the client to
complete full
> recovery of all state. And remember if one client is doing this, it's
likely many
> others are too.
> 
> The spec recommends 90. Why does your server choose a value two orders of
> magnitude lower?

We allow configuration of the lease time, foolishly we don't set a floor. So
someone has set a 2 second lease time, which due to how we actually track
it, is effectively not even one second...

So what I'm looking for is what an absolute minimum might be to set a
minimum value in the config (and I guess I should have been more clear about
that...). Our default (which most people don't touch except in test
environments) is 90.

Frank

---

From: Tom Talpey <[email-masked]>
Date: Thu, 13 Dec 2018 09:02:37 -0500

> that...). Our default (which most people don't touch except in test

Oh, in that case it's not really a constant, based at least on
the things I mentioned before. Personally I'd say server restart
and TCP backoff might make the biggest impacts, followed closely
by server load and the amount of state the client needs to
recover.

Ten seconds? But any specific value, including 90 seconds, can be
playing with fire! Be sure to round up the value internally to
match your implementation, if you allow the administrator to
set N seconds but then implement N-1 or N-2, that would be bad.
Make sure it's *at least* the expected value.

Some might argue "forever" is the better choice btw. Require the
administrator, or some larger watchdog facility, to implement
revocation.

Tom.

---

From: "faibish, sorin" <[email-masked]>
Date: Thu, 13 Dec 2018 14:12:37 +0000

-----Original Message-----
From: nfsv4 <[email-masked]> On Behalf Of Tom Talpey
Sent: Thursday, December 13, 2018 8:31 AM
To: [email-masked]
Subject: Re: [nfsv4] What is a reasonable minimum lease time?

[EXTERNAL EMAIL] 

On 12/12/2018 5:22 PM, Frank Filz wrote:
> reasonable lease time is more like 5 or 10 seconds, that 1 second becomes less significant.
> 

Actually I thnk your lease time is one second, based on the description above. One second (or even two) seems ridiculously short, to me.

The lease time needs to be at least as large as the time needed to connect, establish a session (including authentication), and for the client to complete full recovery of all state. And remember if one client is doing this, it's likely many others are too.

The spec recommends 90. Why does your server choose a value two orders of magnitude lower?
[sf] I can tell you what I saw in several systems in the fileld (VM as well): more than 30 sec. Maybe 90 sec is too much, but 1 sec is too short considering the reason for renew.

./Sorin

Tom.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: [email-masked] (J. Bruce Fields)
Date: Fri, 28 Dec 2018 17:22:36 -0500

> if one client is doing this, it's likely many others are too.

There's a question here which I can't find an explicit answer for in the
spec, which is whose responsibility it is to allow for that time.

There's language recommending that the client allow for propogation
delay to make sure renewals reach the server before the lease expires.

But clearly that's not enough on its own.  If the client sends renews
just in time for them to reach the server, and the server enforces grace
periods of exactly one lease period, then an unluckily timed renewal
won't leave the client with time to reclaim.

Write "L" for the lease time and "M" for the time required to connect
and recover state.  Then you're safe if either the server extends the
grace period up to L+M, or if the client renews as often as L-M.

The former would allow lease periods very close to zero, the latter
would require lease periods of at least M.

Anyway, the current upstream Linux behavior, if I'm reading the code
right:

	- The client renews at intervals of max(2L/3, 5) seconds.  (So a
	  2-second lease definitely won't work reliably.)

	- The server allows lease times between 10s and an hour, with a
	  default of 45 seconds.  The grace period ends L seconds after
	  server start, but may be extended up to 2L if it receives at
	  least one new reclaim each second.  (And may be ended sooner
	  if RECLAIM_COMPLETEs allow it.)

--b.

---

From: David Noveck <[email-masked]>
Date: Sat, 29 Dec 2018 06:21:21 -0500

> spec, which is whose responsibility it is to allow for that time.

I don't see an explcit answer, either for NFSv4.0 or NFSv4.1, so, unless
someone finds this explicitly addreesed, we'll need to figure out how
people should best address this, in the absence of anexplicit assigment of
responsibility.

Note the important difference between v4.0 and v4.1:

   - RFC7530 assumes that the grace period and the lease period are the
   same,  with the implication that determination of the lease period account
   for all the time involved in reconnecting and authenticating, and
   recovering state, including the potential inteference of other recovering
   clients.   Note that these delays are not explcitly noted in the spec,
   although there is a discussion of the time issues involved in setting the
   lease period having to do with getting the client notified in time of a
   breakdown in the connection.
   - RFC5661 allows the grace perod to be longer than the lease period,
   only requiring that it be as long as the lease period.   The implication is
   that the time for the client to do the recovery, once it is aware of the
   need to do so are to be accounted for by raising the grace period
   sufficiently above the lease period.

Overall, since the server sets the lease period and (where allowed to be
different) the grace period, the clear implication is that it is its
responsibility to do so appropriately.   I don't think there is any way
this can be the client's responsibility to deal with especcially as one of
the important considerations in knowing the the recovery time (i.e grace
period minus lease period), is the number of contending recovering clients.

> delay to make sure renewals reach the server before the lease expires.

Yes.

> But clearly that's not enough on its own.

It's enough to deal with the issue of propagation delays, which was the
intention.

It doesn't magically make the whole scheme work, nor does it cure cancer.

> If the client sends renews

Right.   This is a big issue for NFSv4.0, which says these are equal
although there
is no "MUST" with regard to it.

> Write "L" for the lease timeand "M" for the time required to connect
> and recover state.

> Then you're safe if either the server extends the
> grace period up to L+M,

Although this is not indicated expllicitly, that's how I understand the
intention of RFC5661.

> or if the client renews as often as L-M.

I'm not sure how this could be implemented.   How could a client be
expected to determine
M, as it is potentially affected by the number of potential clients.

> The former would allow lease periods very close to zero,

True, but in my mind, that is not a plus.

> the latter would require lease periods of at least M.

Yes, but potentially with very frequest renewals :-(

 >         2-second lease definitely won't work reliably.)

It sound like anything 5 seconds is going to be a problem.

>          least one new reclaim each second.

This deals appropriately with the caseof clients that need t reclaim a lot
of state.
It might need supplementation to deal with the case of hordes of clients
connecting and
authenticating simultaneously.

> (And may be ended sooner if RECLAIM_COMPLETEs allow it.)
The existence of RECLAIM_COMPLETE reduces the downsides of long grace
periods

On Fri, Dec 28, 2018 at 5:22 PM J. Bruce Fields <[email-masked]>
wrote:

> lease
> and

---

From: "J. Bruce Fields" <[email-masked]>
Date: Sat, 29 Dec 2018 17:09:02 -0500

> period minus lease period), is the number of contending recovering clients.

Don't you also need to know how much state in total needs to be
recovered?  The typical server probably doesn't know how much state its
average client has to recover.  I suppose it could set some upper
bounds.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Sat, 29 Dec 2018 22:56:36 +0000

>average client has to recover.  I suppose it could set some upper
True. However the server does know if client(s) are still doing Reclaim type
open/lock operations.

A long time ago I did some testing with the BSD (not yet in FreeBSD) server I had,
where I had 100 clients each holding many locks (can't remember how many) and
then I would reboot the server and the clients weren't able to reclaim all the locks
within the 2minute grace period I had set.

I fixed the problem by having the server extend the grace period until it no longer
saw any Reclaim open/lock operations within the last 1/2 lease duration (or
something like that, I haven't looked at the code for a lonnngggg time).
This seems to work ok, but does require that the initial grace period be long
enough for all clients to reconnect and start doing Reclaim open/lock
operations before the grace period ends.
When I reported this on this mailing list, I think that inspired the addition of
Reclaim_complete to NFSv4.1?

For NFSv4.0, I think the grace period can be greater than the lease duration, if the
server implementor chooses to do so.
(The FreeBSD server uses a default lease duration of 2minutes and this can only
 be changed by editing/recompiling the code so no one can choose absurd values
 for it easily.)

rick