Subject: Mount point crossing & space_avail

From: Peter Åstrand <[email-masked]>
Date: Wed, 20 Mar 2002 16:34:49 +0100 (CET)

Suppose we have a server S with three local filesystems:

/dev/disk1  /
/dev/disk2  /export/  (2 MiB bytes free)
/dev/disk3  /export/pictures/  (2 GiB bytes free)

The /export tree is exported via v4. The /export directory contains one
directory called "documents" (in addition to "pictures", where disk3 is
mounted). 

Client C mounts S:/export on /mnt. On the client, it looks like this:

$ cd /mnt
$ ls
documents  pictures
$ 

User U at client C is planning to copy 500 MiB of data to /mnt/pictures
and first checks if there is enough room for it:

$ df
Filesystem            Size  Used Avail Use% Mounted on
...
S:/export             ...   ...     2M  ... /mnt

User U, which is totally unaware of the servers disk configuration, fsids 
and "All That Jazz :-)" screams and think it's time to buy more disks. U 
cannot now that "pictures" actually is on another filesystem. Or have I 
forgotten something?

Wouldn't it be nice to have some way of querying the server of exported
filesystems, e.g. add the functionality of the MOUNT protocols EXPORT
procedure into v4? Some person asked if there was similiar functionality
in v4 at Cthon, and if I remember it correctly the answer was "No, but
most people will continue to run MOUNTD even with v4". I don't think
that's a good solution.

If v4 had some way of querying the server for exported fs's, an
NFSv4-aware df program could be a bit more intelligent:

$ df
Filesystem            Size  Used Avail Use% Mounted on
...
S:/export             ...   ...     2M  ... /mnt
S:/export/pictures    ...   ...     2G  ... (not mounted)

(Maybe it's a bad idea with a df program fetching information not only by
calling statfs(2) but also by contacting server S over the network, but at
least it would be possible.)

But I suppose an EXPORTS operation in v4 will have to wait until 4.1, 
even if we agree on that it's a Good Thing?

-- 
Peter Åstrand                Telephone: +46-13-21 46 00
Cendio Systems               E-mail: [email-masked]
Teknikringen 3
583 30 Linköping
Sweden

---

From: Brent Callaghan <[email-masked]>
Date: Mon, 25 Mar 2002 14:12:02 -0800

Peter Åstrand wrote:
> :
> cannot now that "pictures" actually is on another filesystem. 

Though this is a general property of UNIX-style filesystems and mounts.
The operating system does a pretty good job at concealing the presence
of mounted filesystems from users.

If an NFSv4 client takes care to remain POSIX compliant and perform
a mount whenever an NFSv4 LOOKUP crosses a server mountpoint, then 
commands like "df" etc will return correct information.

> most people will continue to run MOUNTD even with v4". I don't think

Yes, it's a common request for NFSv4 to support a procedure like
MOUNTPROC_EXPORT from the mount protocol, that'll allow a client
to easily enumerate the list of exported filesystems.  But there
are a few problems with this:

	o Security.  The list is accessible to anyone.  So it may
	  contain the names of confidential filesystems that are
	  invisible to unauthorised users, e.g.

		/export/private/bomb_plot
		/export/private/assasination_targets

	  A v4 server can restrict access to these exports now
	  simply by controlling access to the /export/private
	  directory (even if it's in a pseudo filesystem).

	o Stale information.  The list can become stale.  Currently
	  the automounter uses MOUNTPROC_EXPORT to provide a list
	  when making a /net/<hostname> access.  If the server admin
	  changes the exports while the client has one or more filesystems
	  mounted, there's no easy way to notify of the change.

	o Definition.  If the list describes server exports, what is 
	  an export?  This has never been clearly defined (even for 
	  NFS v2/v3).   We normally consider an export to be an exported
	  filesystem on the server, but it's also possible to export
	  multiple directories within a single filesystem on the server.
	  The closest v2/v3 definition is likely "a directory that has
	  a filehandle available only via the mount protocol", but this
	  doesn't apply to NFSv4.

Having said all that, I think it's useful that an NFSv4 server be able
to "publish" a list of "interesting" directories for those who are
interested in browsing around.  But I'd prefer that this act of publishing
be separate from the protocol.  For instance, you could publish interesting
directories into a network database or name service in the form of an NFS
URL.  The if you  wanted to view a list of mountable directories on an NFS
server, you could just query the database/nameservice for all URLs that
name that server.

	Brent

---

From: Peter Åstrand <[email-masked]>
Date: Tue, 26 Mar 2002 13:24:23 +0100 (CET)

> of mounted filesystems from users.

Yes, but the information should be available if the user or admin requests 
it. 

It's true that the problems I talk about exist with todays automounters as
well, but I think the problem is not as bad as with NFSv4. Automounters
typically uses static maps. If I'm unsure of which filesystems that exist,
I can look at /etc/auto.master, do "ypcat auto.home" or something like
that. The local OS may "hide" things sometimes, but it's even worse if the
server hides things.

What about Windows clients? I assume they have no automount-functionality
and don't care about fsids. Example: S:/export is mapped as d:\. In this
case, it will be impossible to know that d:\documents is another file
system (with another space_avail figure) than d:\pictures, or? If I
right-click on d:\ and want to look at the available space, what will be
presented? space_avail for S:/export i assume? And if I map S:/ (the
pseudo-fs root) to D:\, then we have no idea whatsoever, since the
pseudo-fs does not support the attribute space_avail.

It's probably impossible to present the disk free information through the
normal Windows interface perfectly, but I think NFSv4 should at least make
it possible to write an tool that reliably can display filesystems & free
space.

Basically, I want clients to be able to ask servers: "Which filesystems do
you export?". 

> commands like "df" etc will return correct information.

Yes, but only if all moint points are crossed: The entire exported tree 
must be traversed. 

> 	  directory (even if it's in a pseudo filesystem).

But this problem existed with v2/v3 as well? The operation I want should 
basically return the information from /etc/export and "mount". If the 
admin chooses to use mount points with secret names, then it's the admins 
choice. 

> 	  mounted, there's no easy way to notify of the change.

Why should the client be notified? A tool for listing all exported fs:es 
and fetch information about how much free space they have should contact 
the server directly to fetch fresh information about exports. There is no 
need to cache this information on the client. 

> 	  a filehandle available only via the mount protocol", but this

It was a mistake from me to say that I want the old MOUNTROC_EXPORT to be
incorporated into v4. This is not true. What I want is a operation that
returns all directories in the exported tree where fsid changes. A simple
definition.

> server, you could just query the database/nameservice for all URLs that

Yes, that is an option. Do you think there are other properties than fsid 
that would lead to an "interesting" directory? I can't think of any right 
know. 

My preference would be to limit "interesting directories" to just
"directores where fsid changes" and adding an operation for returning such
a list to the protocol.

Btw, how many clients have implemented "mirror-mounting" on mount point
crossing?

---

From: Mike Eisler <[email-masked]>
Date: Tue, 26 Mar 2002 06:29:30 -0800

> > commands like "df" etc will return correct information.

I hope that all UNIX-based clients are doing this now. Otherwise,
very bad things will result.

> But this problem existed with v2/v3 as well? The operation I want should

Just because v2/3 had security holes is no reason for
v4 to propogate them.

> returns all directories in the exported tree where fsid changes. A simple

That is quite possibly an infinite, or at least very long
lsit. Consider NFS servers that re-export data from NFS
automounted filesystems, AFS, SMB, ftp, http. etc.

> > server, you could just query the database/nameservice for all URLs that

Maybe we need an snmp mib for nfsv4.

	-mre

---

From: Mike Eisler <[email-masked]>
Date: Tue, 26 Mar 2002 07:36:20 -0800

[email-masked] wrote:
> 
> > Date: Tue, 26 Mar 2002 06:29:30 -0800
> > From: Mike Eisler <[email-masked]>
> >
> Is it possible for Unix clients to bound the amount of state they must
> keep given this approach?
> 
> Suppose someone implements a file server that returns a different
> fsid4 for every file.  Even disregarding things like df, a Unix

Then few UNIX clients will use that server, hopefully its market
lifetime will tend to zero. But then again, there's a well
known NFS server on the market that doesn't correctly
support close-to-open semantics, and it simply won't die, so
I could be wrong about the market.

> implementation would need to allocate a unique st_dev value for each
> fsid4 in the file system, and then keep track of the mapping from
> fsid4 -> st_dev so as to keep results of stat/fstat system calls
> consistent.  The current way of doing this is with mount points--a
> fairly heavy weight data structure.  So okay, maybe you could devise
> something much lighter weight, but since you can never evict these
> mappings, there's still the question of what to do when you're using
> too much memory for them.

In that event, the client's NFSv4 filesystem has at least two possible
strategies:

	1. return an error to the application that is attempting to
		access the directory that triggered the failed mount
		attempt.
		
	2. reclaim the memory for a mount no longer in use. If all
		are in use, go back to strategy 1.

> The reason this is important is that something like NFSv4 should be
> useful for getting data from servers you don't trust.  For example, we

This is analagous to the "/net" automounter mount point. There can
be at least one mount for every NFS server your client can connect
to. So this problem, if it exists, needs to be addressed, regardless
of the presence of NFSv4 support.

> like to point our web browsers at servers we don't entirely trust,
> because we nonetheless are ineterested in the content.  Web servers
> should not be able to crash your browser.  Similarly, you might want
> to mount a file system to look at data on a server you don't entirely
> trust, and the server should not be able to crash your kernel.

If there is a strategy 3 for the client, it should not include
crashing the kernel.

	-mre

---

From: Peter Åstrand <[email-masked]>
Date: Tue, 26 Mar 2002 17:29:49 +0100 (CET)

> Just because v2/3 had security holes is no reason for

True, But as I said, I don't consider this a security problem. To me, file 
system root directories are "entry points" and I don't see the need for 
making it possible to embed secrets in such names. 

Just an idea: Wouldn't it be possible for the server to actually check if
the user requesting GETEXPORTEDFS'S has access to all paths in the result?  
The server could try LOOKUP on the paths before returning them. Wouldn't
that work? In that case, we could even satisfy those people that wants to
embed secrets in paths.

> automounted filesystems, AFS, SMB, ftp, http. etc.

I doubt that many servers will return more than a hundred moint points or
so. In some situations it is impossible to even determine all moint
points, true, but then the server can just return NOTSUPP. Or, return
correct data for moint points it is certain about and set a flag
indicating a hierarchy of "unknown" filesystems.

Clearly, some odd servers cannot/don't like to support the operation I
suggest, but I think most can without too much trouble. 

If some people feels bad about having an operation that sometimes return
NOTSUPP (we have lots of them already, but anyway...), does it make more
sense to add a new recommended attribute instead?

> Maybe we need an snmp mib for nfsv4.

That would probably be useful, but to me it sounds crazy that one must 
depend on SNMP, LDAP or something like that just to get the list of 
exported fs's and be able to check where I can find some free disk space.

---

From: Eric Werme USG <[email-masked]>
Date: Tue, 26 Mar 2002 12:05:14 -0500

> list. Consider NFS servers that re-export data from NFS
   
   I doubt that many servers will return more than a hundred mount points or
   so.

On our production systems, we export each user's directory separately so
that we can limit access to particular clients, e.g.:

% showmount -e | grep werme
/share/wasted/usr/staff6/werme      Everyone

% showmount -e | grep someone-else
/share/alpha/usr/staff6/abrooks     nodea nodeb nodec
   
% showmount -e | wc
       243      1052     15132

If our mountd were brighter, I suppose it could return just the subset
of things that have a chance of being mounted, but even that can be
substantial for various netgroups and things:

% showmount -e | egrep 'Everyone|production|zk3' | wc
        71       349      4990

The actual number of file systems mounted on this 3 node cluster is:

% df -t advfs | wc
        72       433      7859

One problem with customers who export hundreds of file systems is that they
tend to have gold service contracts and are vocal.  Programming for
typical system is perilous.  :-)

	-Ric Werme

---

From: Erez Zadok <[email-masked]>
Date: Tue, 26 Mar 2002 19:10:32 -0500 (EST)

> that we can limit access to particular clients, e.g.:

BTW, I'm sure you know that there are some exploits possible when exporting
a subdir or a file system---where the exploit allows you to get access to
the rest of the f/s.  IIRC, Linux's knfsd couldn't get this working
securely.  But, my information may be old.

Erez.

---

From: Erez Zadok <[email-masked]>
Date: Tue, 26 Mar 2002 19:20:58 -0500 (EST)

> exported fs's and be able to check where I can find some free disk space. 

Didn't we discuss the possibility of SNMP support in the past?  Was anything
decided then?  If SNMP support becomes part of NFSv4, and it would already
define many things one can query, then it may be a small incremental cost to
add support for more SNMP queries (even w/o a formal MIB update).  It seems
to me that SNMP is a "cheap" way of adding some (read-only) functionality to
NFS w/o the overhead of a protocol revision.

Still, having to depend on SNMP for NFS will be a pain for SAs.

So suppose an NFSv4 server can and wants to provide a way to return a list
of exported file systems back to a client?  Does that demand a protocol
update with what we have now in rfc3010?  Is there any way through the
various extensibility mechanisms to support this, or will the client+server
have to be modified to understand a special protocol extension?

Although I don't like the whole idea of ioctl(2), it is a cheap way to
extend user/kernel functionality.  Do we need to consider a similar call
(NFS4_IOCTL) in a future protocol revision?

Erez.