Subject: [nfsv4] Question about OpenOwner table overflow

From: Vitaliy Gusev <[email-masked]>
Date: Wed, 1 Feb 2012 13:15:45 +0400

Hello!

During reading rfc3530 and rfc5661 I didn't find any mentions about 
nfs-server behaviour if client has forgotten about opened file (client 
alive and sends RENEW on time).

So openowner, stateid-s entries on server can grow endlessly.

Am I missing something?

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 1 Feb 2012 13:35:05 +0000

> So openowner, stateid-s entries on server can grow endlessly.

This is why OPEN is allowed to return NFS4ERR_RESOURCE which, in this
instance, is the equivalent of returning the POSIX error "EMFILE". It is
up to the server to determine how many simultaneous open files it will
allow the client, and to enforce that limit once it is hit.

Cheers
  Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Vitaliy Gusev <[email-masked]>
Date: Wed, 1 Feb 2012 18:07:57 +0400

> allow the client, and to enforce that limit once it is hit.

Ok. One client can open/forgot 1000 files. 1000 clients can open/forgot 
1M files. It looks like good way for DOS attack.

Why do not require to update a OPEN in lease time.

---
Vitaliy Gusev

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 1 Feb 2012 14:34:23 +0000

>1M files. It looks like good way for DOS attack.

Most likely the server would have a ceiling on how many open owners it
would allow altogether.

>_______________________________________________
>nfsv4 mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Vitaliy Gusev <[email-masked]>
Date: Wed, 1 Feb 2012 18:56:01 +0400

> Most likely the server would have a ceiling on how many open owners it

Exactly! Clients can be good and bad (no close).
Imagine that you are good client. You tries to connect to server and
open file. You get ERR_RESOURCE always, because guilty clients can't be
corrected (forced close).

Once again, you want to get service but you can't do it because
server relies on bad clients.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 1 Feb 2012 15:17:16 +0000

> 1M files. It looks like good way for DOS attack.

As Tom said, the server is responsible for managing its full set of
resources.

> Why do not require to update a OPEN in lease time.

That would make the protocol even more chatty than it already is and
wouldn't really solve the problem. If you have 1000 clients, you can
easily keep them pinging that server with 1000 files each.

Note that if the problem is that of abusive clients that never close
files in situations where they should, then the server can always
declare a reboot and get them to reclaim those opens that are actually
in use.
Alternatively, change the authentication requirements and only allow
trusted clients...

NetApp

---

From: Vitaliy Gusev <[email-masked]>
Date: Thu, 16 Feb 2012 15:40:40 +0400

> declare a reboot and get them to reclaim those opens that are actually

Can a server simulate the reboot only for those "bad" clients whereas 
other clients will work with the server in normal mode ?

> Alternatively, change the authentication requirements and only allow

I meant not security problem. Our world is not perfect. And NFS clients 
and NFS servers are too. So servers should have basic protection 
mechanism against resources leaking-forgetting, locks forgetting, etc.

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 16 Feb 2012 06:55:28 -0500

On Thu, Feb 16, 2012 at 03:40:40PM +0400, Vitaliy Gusev wrote:
> Can a server simulate the reboot only for those "bad" clients
> whereas other clients will work with the server in normal mode ?

I can't see why that wouldn't work.

--b.

> I meant not security problem. Our world is not perfect. And NFS
> clients and NFS servers are too. So servers should have basic
> protection mechanism against resources leaking-forgetting, locks
> forgetting, etc.
> 
> 
>

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 16 Feb 2012 13:32:15 +0000

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Thursday, February 16, 2012 6:55 AM
> To: Vitaliy Gusev
> Cc: Myklebust, Trond; [email-masked]
> Subject: Re: Question about OpenOwner table overflow
> 
> > >declare a reboot and get them to reclaim those opens that are
> > >actually in use.
> >

The main problem would be in enforcing the grace period... You'd have to DELAY all potentially conflicting OPEN or LOCK requests from other clients until the client that are in grace have finished recovering.

Cheers
  Trond

---

From: Rick Macklem <[email-masked]>
Date: Thu, 16 Feb 2012 10:48:50 -0500 (EST)

Trond Myklebust wrote:
> > > whereas
> The main problem would be in enforcing the grace period... You'd have
> to DELAY all potentially conflicting OPEN or LOCK requests from other
> clients until the client that are in grace have finished recovering.
> 
Also, you'd have to be careful how you handle the edge conditions mentioned
in Sec. 8.6.3 of RFC3530 (they're in the newer specs, but I don't have
the reference handy). I haven't thought about it, so I don't know if this
is more difficult when only certain clients "see it rebooted". 

At the least, I'd suggest that if a server detects a client being an
openowner hog, that it log that fact, since at least the newer versions
of Linux and FreeBSD client have been patched to use ReleaseLockOwner to
avoid this. (I don't know where the other clients are w.r.t. this.)

rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 16 Feb 2012 10:56:29 -0500

> is more difficult when only certain clients "see it rebooted". 

If the client doesn't reclaim then you'd need to log that fact to stable
storage before starting to grant potentially conflicting locks.  I don't
think that's too different from what you'd normally do when a client
expires.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 16 Feb 2012 11:39:55 -0500 (EST)

> is more difficult when only certain clients "see it rebooted".
Oh, and another one that might get tricky...
- You'd need to generate clientids and stateids that somehow refer
  to the "current boot instance" for both the clients that "saw a reboot"
  and those that didn't, so that you can reply NFS4ERR_STALE_CLIENTID/STATEID
  when necessary.
  An example of this from a recent thread:
  - The FreeBSD server determines whether it should reply NFS4ERR_EXPIRED
    vs NFS4ERR_STALE_CLIENTID/STATEID when it doesn't have a record for
    the clientid by looking at the clientid/stateid bits to determine
    if it is for the current "boot instance". If I were to "partition"
    the current boot instance bit space in the clientid/stateid to handle
    "only some clients saw reboot", then I'd run out of bits.
--> You might find you have to keep records for even expired clients until
    the server reboots for real and forgets everything, to handle the
    NFS4ERR_EXPIRED case.

I'm not saying you can't make it work, but there might be a few "alligators
in the swamp" that could make it interesting:-) rick

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 16 Feb 2012 11:42:19 -0500

> in the swamp" that could make it interesting:-) rick

Dumb question, probably showing that I've ignored whole swaths of email
traffic about courtesy locks--but are clients expected to treat STALE
and EXPIRED much differently?

--b.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 16 Feb 2012 18:18:21 -0500 (EST)

J. Bruce Fields wrote:
> > > > > > >Note that if the problem is that of abusive clients that
> > > > > > >never
> > > > > > >close
> > > the reference handy). I haven't thought about it, so I don't know
> > > if
> > > this
He he. Well, I think the handling of NFS4ERR_EXPIRED is still an open
question that David Noveck has been trying to nail down. You can look in
the recent thread or you can look at an old thread in the mail archive,
which is back in Oct. 2004 with NFS4ERR_ADMIN_REVOKE in the subject header.

I've posted with what the FreeBSD client currently does when it sees
NFS4ERR_EXPIRED, but I don't claim it is either correct nor the best
approach. (My understanding is that a recovery from NFS4ERR_EXPIRED
cannot use Lock with reclaim == true or Open with CLAIM_PREVIOUS, since
there isn't normally a grace period. As for whether or not it should
be preceded by SetClientID/SetClientIDConfirm seems to be in some doubt,
but the FreeBSD client does it. The FreeBSD client also never tries to
recover byte range locks, since it cannot know if a conflicting lock
was acquired/released during the network partitioning. It only tries
to recover Opens (all FreeBSD client opens use ACCESS_DENY_NONE) via
a CLAIM_NULL.

rick

---

From: Nico Williams <[email-masked]>
Date: Thu, 16 Feb 2012 17:28:01 -0600

> declare a reboot and get them to reclaim those opens that are actually

If apps leak file descriptors then the clients will still recover all
those opens, and the server once again runs out.

> Alternatively, change the authentication requirements and only allow

Yes.  And make sure you have no apps that leak file descriptors.

Nico
--

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Tue, 21 Feb 2012 04:37:38 +0000

> > > > >Note that if the problem is that of abusive clients that never
> > > > >close files in situations where they should, then the server can
> > > > >always declare a reboot and get them to reclaim those opens that
> > > > >are actually in use.
> > > >
> Also, you'd have to be careful how you handle the edge conditions
> mentioned in Sec. 8.6.3 of RFC3530 (they're in the newer specs, but I don't
> have the reference handy). I haven't thought about it, so I don't know if this
> At the least, I'd suggest that if a server detects a client being an openowner
> hog, that it log that fact, since at least the newer versions of Linux and
> FreeBSD client have been patched to use ReleaseLockOwner to avoid this. (I
> don't know where the other clients are w.r.t. this.)

Maybe, I'm not reading the spec right. But, ReleaseLockOwner frees Lock Owners only. Is this part of some errata?

Trond can correct me, but Linux client does not issue ReleaseLockOwner for open owners. What I'm aware of is that the new Linux client optimizes the generation of a new owner for every OPEN operation, by sharing an open owner across multiple opens from the same NFS creds. How does FreeBsd do it?

- Pranoop

---

From: Rick Macklem <[email-masked]>
Date: Mon, 20 Feb 2012 23:45:23 -0500 (EST)

Pranoop Erasani wrote:
> > openowner
> Maybe, I'm not reading the spec right. But, ReleaseLockOwner frees
> Lock Owners only. Is this part of some errata?
> 
> Trond can correct me, but Linux client does not issue ReleaseLockOwner
> for open owners. What I'm aware of is that the new Linux client
> optimizes the generation of a new owner for every OPEN operation, by
> sharing an open owner across multiple opens from the same NFS creds.
> How does FreeBsd do it?
> 
You are correct. I mistakenly thought you were discussing the situation
where:
- parent opens file and holds it open for a very long time
  - child of this process does byte range locking
    - if it doesn't do a ReleaseLockOwner, the server can't discard the
      lock_onwer4.
This problem was a bug in old versions of both the Linux and FreeBSD
clients, but both are now patched to do ReleaseLockOwner Ops.

I had forgotten (it was related to a post that came along quite a
while ago) you were asking about a hypothetical client that fails
to close opens. (I don't know of an extant client that fails to close
opens, but there certainly could be a busted one someday, I suppose.)

Sorry about the confusion, rick

---

From: Nico Williams <[email-masked]>
Date: Tue, 21 Feb 2012 00:07:57 -0600

There certainly exist apps that leak file descriptors...  It is
something that valgrind checks for, for example.

But one might as well be concerned about apps that "attempt" to DoS
the server in many other ways... like trying to fill up the
filesystems.  (I remember a situation in the mid to late 90s where a
few hundred or so clients all had the same cron job to find and remove
.nfs files... and they all ran at the same time... the poor file
server -an E4K, which at the time was a fairly large system- couldn't
keep up.)

Nico
--