Subject: READDIR attributes ambiguity

From: David Mazieres <[email-masked]>
Date: Mon, 6 Mar 2000 17:08:08 -0500 (EST)

The NFS4 draft spec does not make explicit which attributes READDIR
should return for a mount point.  For example, suppose a server with
two disks has the following directories:

  /      fsid=1, fileid=2
  /a     fsid=1, fileid=3
  /b     fsid=2, fileid=3   (mounted over /b fsid=1, flieid=4)

When requesting the fsid and fileid attributes from READDIR, Which
fsid/fileid pair should READDIR return for file name "b".

Implementers would probably prefer to return fsid=1, fileid=4, because
this is more efficient in many cases.

The spec would imply that the correct answer is fsid=2, flieid=3,
because technically that is correct, and it never says READDIR can
return anything else.

In fact, though, if you implement what I think the spec requires, it
becomes difficult to implement functions like the ANSI C getcwd()
efficiently.  Moreover, most implementations of getcwd (which are
optimized to take advantage getting the fileid from OS-specific
readdir extensions) will need to be changed before they work on NFS4.

In the example above, getcwd() might not work from within /a, because
the client OS can never see the hidden fileid 4 and therefor cannot
return a fileid unique to fsid=1 for the directory entry /b.

David

---

From: Mike Eisler <[email-masked]>
Date: Mon, 6 Mar 2000 15:08:58 -0800 (PST)

> because technically that is correct, and it never says READDIR can

Yes that's the correct answer. To do otherwise is inconsistent with
the mount point crossing of all the other procedures that deal with
path names.

> readdir extensions) will need to be changed before they work on NFS4.

I doo have been concerned about this, but I don't think it is impossible to
implement. 

> return a fileid unique to fsid=1 for the directory entry /b.

But the client OS can see that the fsid changes. If nothing else, the client
file system can fake the inode # in the readdir results when the entry is a
mount point.

	-mre

---

From: David Mazieres <[email-masked]>
Date: Mon, 6 Mar 2000 18:18:49 -0500 (EST)

> Date: Mon, 6 Mar 2000 15:08:58 -0800 (PST)
> From: Mike Eisler <[email-masked]>
> 
> >   /      fsid=1, fileid=2
> >   /a     fsid=1, fileid=3

What fake inode # should it use.  Are any fileid's reserved for the
client (like maybe 0)?  The problem is the client cannot pick a unique
fileid to make up, because, in the example, it doesn't even know about
the number 4 which would be guaranteed to be unique.

David

---

From: Mike Eisler <[email-masked]>
Date: Mon, 6 Mar 2000 15:35:34 -0800 (PST)

> the number 4 which would be guaranteed to be unique.

Since the purpose is give getcwd a fileid that is different from that of the
/b, any fileid that doesn't conflict with the readdir results should suffice. The
getcwd() library code I'm looking at as this snippet:

 		if (nd.st_ino != dir->d_ino) {
                        (void) closedir(ndirp);
                        goto do_mount_pt;
                }

Personally, I think comparing the inode# from readdir to that from stat
is a extremely crude. 

	-mre

---

From: David Mazieres <[email-masked]>
Date: Mon, 6 Mar 2000 22:43:56 -0500 (EST)

> Date: Mon, 6 Mar 2000 15:35:34 -0800 (PST)
> /b, any fileid that doesn't conflict with the readdir results should
> suffice.

But you might not get the fileid for /b until a subsequent readdir.
Currently, the only way to guarantee that a fileid you generate will
not conflict with another fileid in the same directory would be to
read the entire directory first.  It would be nice to support existing
implementations of getcwd without committing to unbounded storage in
the kernel.

> The

Well, it's also much more efficient.  It saves a lot of system calls,
and on some file systems it saves a lot of RPCs.  Obviously NFS3 and 4
will cost you fewer RPCs than NFS2 (at least in theory--several OSes
prime the attribute cache but not the name cache with READDIRPLUS).
Nonetheless, people will still want getcwd() to work fast on NFS2.

There's also, of course, the question of efficiency at the server,
though that is probably less of an issue.

David

---

From: Brent <[email-masked]>
Date: Tue, 07 Mar 2000 00:32:08 -0800

> implementations of getcwd without committing to unbounded storage in

Perhaps an easier way to resolve this would be to have a "mounted on"
fileid attribute, e.g. call it "mfileid".

POSIX clients could then ask READDIR to return both "fileid" and
"mfileid"
attributes.  The "mfileid" would be returned in response to getdents()
calls
on UNIX clients, and the "fileid" would be cached along with other file
attributes for the directory entry, i.e. a subsequent "stat()" call
on the client would return the "fileid".

	Brent

---

From: Mike Eisler <[email-masked]>
Date: Tue, 7 Mar 2000 07:18:22 -0800 (PST)

> But you might not get the fileid for /b until a subsequent readdir.

I don't undnerstand. If a reader is not on the server's /, you'll get the
fileid of what is mounted on /b.

> implementations of getcwd without committing to unbounded storage in

Someday, and that day may even be today with W2k, non-UNIX systems are going
to mount drives in arbitrary places, and aren't necessarily going to adhere
to UNIX semantics. The time is nigh for getcwd to adapt.

> Well, it's also much more efficient.  It saves a lot of system calls,

You can achieve the same effect by looking at the st_dev of the directory and
the directory's "..".

As Brent says, a "mounted on fileid" attribute would be another way to solve
this. NFSv4 testing is going at connectathon this week, and it would be
interesting to get experience before we rush in and add the attribute.

	-mre

---

From: Mike Eisler <[email-masked]>
Date: Tue, 7 Mar 2000 07:21:18 -0800 (PST)

> fileid of what is mounted on /b.

Let's try that again:

If a readdir is on the server's /, you'll get the fileid of what is mounted on
/b.

---

From: David Mazieres <[email-masked]>
Date: Tue, 7 Mar 2000 11:11:01 -0500 (EST)

> Date: Tue, 7 Mar 2000 07:18:22 -0800 (PST)

That's correct, and fileids are not unique across fsid's.  Software
makes two assumptions:

   * st_ino's are unique within a given file system
   * all files in a directory are in the same file system

An NFS4 client implementation cannot make these guarantees in constant
space.  (It could maybe read the whole directory and make up unique
fileids where the fsids do not match, but this will not work well on
large directories.)

> to UNIX semantics. The time is nigh for getcwd to adapt.

Can't you already mount drives in arbitrary places in UNIX?  Certainly
it's easy for the server to make up fileid's in software for each
mount point, since the server knows about all the mount points.

> You can achieve the same effect by looking at the st_dev of the directory and

Indeed, this is what's going on.  I believe the common algorithm for
getcwd is as follows:

     stat (".", &sb);
     stat ("..", &sb2);

     if (sb.st_dev != sb2.st_dev) {
       /* Different file systems, so lstat every file in .. */
     }
     else {
       /* Same file system, look for d_ino matching sb.st_ino */
     }

> interesting to get experience before we rush in and add the attribute.

Well, whether or not the problem comes up depends on how you mount
things.  This is a low probability bug, depending on inumber
allocations, but will be really annoying when it occurs.

To see the bug, you would need to need to mount a file system in which
the inumber of the root equals the inumber of another sibling
directory in the parent file system.

David

---

From: Mike Eisler <[email-masked]>
Date: Tue, 7 Mar 2000 08:23:39 -0800 (PST)

> fileids where the fsids do not match, but this will not work well on

And by the way, the algorithm that getcwd() uses doesn't scale on large
directories, with or with out Brent's "fileid on mounted dir." attribute.

getcwd() needs to enter the 21st century.

> mount point, since the server knows about all the mount points.

Yes you can, and UNIX local file systems will preserve the semantics that
getcwd() wants. I'm saying that non-UNIX systems might not, and if so,
then what? The "fileid on mounted dir." attribute buys you absolutely
nohting, especially since non-UNIX systems often routinely fake fileids
for READDIR and GETATTR.

> directory in the parent file system.

In UNIX land, isn't the fileid of root always equal to 2, and the fileid
of any other directory not euqal to 2?

	-mre