Subject: Re: [reiserfs-list] Directory cookie issues

From: Eric Werme USG <[email-masked]>
Date: Mon, 5 Feb 2001 10:14:56 -0500 (EST)

Xuan Baldauf wrote:

>Maybe a directory format plugin would be desirable. People needing
>stable NFS use the hash format (with the various hash
>algorithms). People needing fast access to their thousands of
>subdirectories use alternative formats (e.g. stem compression,
>everything you like) at the cost of being not fully NFSv2 compatible
>(and having a question mark at NFS in general). The cases where both
>needs (NFS and large directories) collide are much more rare than the
>cases where the current hash format and large directories do collide.

I think you lost me here.  Do you mean a "protocol plugin" that both
client and server would use?  Do you mean something on just the server
that both VFS and NFS would use?  Do you mean something that reiserfs
would use to allow admins to configure the file system to their
application?

>But be cautious, if you implement directory format plugins, plugins
>for other things (like stat data emulator), etc., we can end up in
>SuperFS. *grin*

What is "stat data emulator"?  Are there issues with that and NFS V4's
named attributes?

Hans, I think the sooner you or someone can describe exactly what
reiserfs needs from NFS V4, the more concise the Emails will be!

	-Ric Werme

---

From: Xuan Baldauf <[email-masked]>
Date: Mon, 05 Feb 2001 18:28:21 +0100

> would use to allow admins to configure the file system to their

Sorry Ric, the text aboce and below were intended for the ReiserFS
developers. A directory format plugin would decide how to internally store
directory data within a filesystem. This primarily does not have to do
with NFS, only secondary (not every directory data format is suitable for
current NFS).

> What is "stat data emulator"?

The stat data emulator is a little program which to with the stat() call
for a file is handed over. Many applications which create millions of
files do not need stat data, so instead of storing stat data for every
file, you just store some information into the directory of the file that
every file in this directory does not have stat data, it is created by the
stat data emulator program. This has nothing to do with NFS, it was
intended for the ReiserFS developers.

>  Are there issues with that and NFS V4's
> named attributes?
>

Xuân.

---

From: Hans Reiser <[email-masked]>
Date: Mon, 05 Feb 2001 23:12:36 +0300

> reiserfs needs from NFS V4, the more concise the Emails will be!

Opaque cookies with variable length that can be at least the size of a maximum
allowed filename would be great.  Eric, you need to add a field specifying the
length of the opaque item, yes?

Do I understand correctly that you already plan to allow the filesystem to not
find files by the file handle if the dcache has discarded the inode from RAM,
and so we can effectively require NFS to give us the pathname again if the
layers above us have discarded the dcache entry that allows mapping file handle
to key of file? Do I understand correctly that NFS v4 doesn't see inode numbers,
it just sees file handles, and so if a few keys can't fit into a file handle
(which is longer than the average filename) I can just require access by
pathname every time for those filenames?

If I understand correctly, then I think we only need variable length cookies. 
What will be the length of a filehandle and will it be variable in v4?

Thanks Eric,

Hans

---

From: Felix von Leitner <[email-masked]>
Date: Mon, 5 Feb 2001 21:57:27 +0100

> length of the opaque item, yes?

Argh, NO!

  client 1		client 2
  ------------------------------
  open("foo") -> fd 2
			rename("foo","bar")
  read(2,...)

Once the file is open, the file name should not be involved.

We need a scheme that allows finding a file without the name, even when
it has been deleted and the NFS server just rebooted.

Felix

---

From: Hans Reiser <[email-masked]>
Date: Mon, 05 Feb 2001 23:36:34 +0300

> > length of the opaque item, yes?
I don't have a problem with providing a means to find a file regardless of
whether it has been renamed, so long as it is not a fixed length int that is
used to find it.  This is a completely different topic from cookies though.

Hans

---

From: Michael Dilger <[email-masked]>
Date: Mon, 05 Feb 2001 13:09:22 -0800

Felix von Leitner wrote:
[...]
> Once the file is open, the file name should not be involved.

This is just a quick reiteration of Felix's point.  Many wonderful
security holes surface when there are sequential references to a
handle which can be redirected to a different object by parties
outside of the operation.  A BAD thing that has been a recurring
security hole in setuid programs is:

	client 1			client 2
	--------			--------
	access("foo") --> "access ok"	
					symlink foo to /etc/shadow
	open("foo) with privledges.

Disclaimer: I am only speaking to this specific point, and not to
the general discussion, which I am quite unfamiliar with at the
present.

-Mike

---

From: Hans Reiser <[email-masked]>
Date: Mon, 05 Feb 2001 23:49:02 +0300

I think Xuan is right on target in his comments below.

Hans

Xuan Baldauf wrote:
> > it has been deleted
> That's reasonable.
> > and the NFS server just rebooted.
> 
> That's not reasonable. How should the NFS server know when to free the unlinked file
> if no one is accessing it? (It apparently does not know after reboot that there was
> an NFS access to that file.) The
> unlinked-but-accessed-and-still-existing-after-unclean-shutdown-file-problem is to
> be solved by a special "to-be-freed" directory, whose contents are really freed
> after journal replay but before finishing the mount() call. This is incompatible
> with your request to let an unlinked file be accessible by NFS after reboot. Or
> should the FS write to disk that a file has been opened by NFS?
> 
> >
> >
> > Felix
> 
> XuБn.

---

From: Xuan Baldauf <[email-masked]>
Date: Mon, 05 Feb 2001 22:12:52 +0100

That's reasonable.

Xuân.

---

From: Eric Werme USG <[email-masked]>
Date: Mon, 05 Feb 2001 16:53:30 -0500

Eric Werme USG wrote:

   Opaque cookies with variable length that can be at least the size
   of a maximum allowed filename would be great.  Eric, you need to
   add a field specifying the length of the opaque item, yes?

No, the "opaque foo<>" is RPC meta language that implies that.  The over the
wire data would be <32 bits of big endian byte count><that number of
bytes rounded to the next multiple of 32 bits>.

   Do I understand correctly that you already plan to allow the
   filesystem to not find files by the file handle if the dcache has
   discarded the inode from RAM, and so we can effectively require NFS
   to give us the pathname again if the layers above us have discarded
   the dcache entry that allows mapping file handle to key of file?

I'm just getting back to dealing with V4 issues after a long period of
not having enough time for V4.  V4 has a volatile file handle mechanism
that I think will do what you need.  It will support a system that returns
a file handle for a file, but after some time stops working.  If the NFS
server or physical file system can cache a file handle to file mapping
that keeps the file handle working for a while, e.g. long enough to read
or write it, preferably longer, then I think it's what you want.  The ban
on file handle reuse "for a long time" remains, but I think that's an
easy implementation issue.  See RFS 3010, look for volatile file handles.

 Do
   I understand correctly that NFS v4 doesn't see inode numbers, it
   just sees file handles, and so if a few keys can't fit into a file
   handle (which is longer than the average filename) I can just
   require access by pathname every time for those filenames?

Every time is a problem.  Need a cache.  

File handles will be
    const NFS4_FHSIZE = 128;
    typedef opaque  nfs_fh4<NFS4_FHSIZE>;
i.e., upto 128 bytes.

However, some of that space needs to be used by servers to guarantee
non-reuse and whatnot.

As for changing the cookie definition, I have to leave it to those who have
been more active than me.  Okay guys, the sun is up in California.  If you
have electricity, do you have any comments?

	-Ric Werme

---

From: Hans Reiser <[email-masked]>
Date: Tue, 06 Feb 2001 00:31:15 +0300

> Every time is a problem.  Need a cache.

Ok, this is fine too, especially if the server implementation is nice enough to
hide it from me.

---

From: Felix von Leitner <[email-masked]>
Date: Mon, 5 Feb 2001 23:34:55 +0100

> I think Xuan is right on target in his comments below.

Please don't jump to conclusions; read the NFS standard first.

NFSv2 explicitly states that file handles must survive a server reboot.

The standard is a mess.  Agreed.
But its requirements are well-defined.

The problem is that you can't really know when the file really is
deleted because clients are allowed to reboot and not finish
transactions.  AFAIK this is handled by Sun NFS by simply refusing to
delete files that are still open and renaming the files to .NFS[number].

The admin was then expected to run a cron job that would look for an
delete these files on the NFS servers.

Felix

---

From: Brent <[email-masked]>
Date: Mon, 05 Feb 2001 14:46:50 -0800

> have electricity, do you have any comments?

Certainly NFSv4 could be more flexible in allowing READDIR cookies to
be bigger than the current limit of 64 bits, but that still leaves a 
problem of implementation.

The READDIR cookie doesn't just sit around in the client's kernel.
It goes back to the app in the getdents structure as the "directory offset"
to be used in subsequent getdents calls.  In the past this has been a 32 bit
offset.  Modern 64 bit UNIXes support a 64 bit getdents directory offset.

The NFS v3 protocol extended the cookie size from the 32 bits of NFS v2
to 64 bits.  That caused a problem for 32-bit OS's because they couldn't
handle big directory offsets from 64 bit servers.

The problem doesn't get any easier with cookies bigger than 64 bits.
Even a 64 bit OS won't handle those unless the offset in the getdents
structure is changed to be some variable length array of bytes rather
than an integer.  What kind of offset does Linux use in getdents ?

	Brent

---

From: Trond Myklebust <[email-masked]>
Date: 06 Feb 2001 12:09:28 +0100

>>>>> " " == Brent  <[email-masked]> writes:

     > The problem doesn't get any easier with cookies bigger than 64
     > bits.  Even a 64 bit OS won't handle those unless the offset in
     > the getdents structure is changed to be some variable length
     > array of bytes rather than an integer.  What kind of offset
     > does Linux use in getdents ?

Currently we just pass the cookie directly to getdents. That's a
pretty lame thing to do (violating POSIX and all), but it's done that
way for historical reasons.

I have now prepared a patch that sets up a cookie dictionary and
creates signed 32-bit offsets for use as getdents/lseek offsets. There
are still some issues of garbage collection to work out, but I hope
that'll ready for inclusion into the official kernel RSN.

Cheers,
  Trond

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 6 Feb 2001 13:43:08 -0600

> that'll ready for inclusion into the official kernel RSN.

Will the size of this new cookie directory grow with the size of the
real directory (unbounded)?  How does the client know when to free the
cookie directory (or is this issue included in the garbage collection
you mention)?

-- 

- Spencer -