Subject: RE: [nfsv4] Fw: Global namespace and multiple file access protocols

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 4 Jan 2004 04:31:24 -0800

> Let's  be clear here.  While we are discussing that a possible future  
> spec should say about how a client might
> get to  the initial location for a file system (direct LDAP access or referrals 
> from v4  servers), the question of
> how  you deal with a MOVED error is clear in RFC3530.  The client is supposed 
> to  get fs_locations from the
> server  and use that to find the new location of the file system.  If it does  
> something else, it is violating the
> spec. 

Okay it's good to have you state this as I had not been assuming this. I assume we would apply this rule to any protocol capable of issuing ERR_MOVED, i.e. CIFS and NFS V4. 

I'm only capable of talking about v4.  I don't know anything about what rules CIFS has.  As I understand the process, Microsoft builds clients, they do what they do, and servers adapt, making the promulgation of any rules we might decide on rather irrelevant.
 
Now how do I extrapolate this to walking down a tree in the global namespace. If we walking down this global namespace with a 'dumb' client aren't we getting ERR_MOVED whenever we cross a server boundary, i.e. if we are walking down a/b/c/d and a/b/c are virtual on a root server and d is a real FS on another server we will get ERR_MOVED when we lookup d - correct ? Hence it would be illegal for any client to do anything in this situation except get the fs_locations from the server - correct ? 

I would say it must get fs_locations from the server.  It may do other things if that doesn't work, e.g. the server is down. 

Now we're saying a smart client may walk down a/b/c/d on the ldap server and then directly access the NFS server exporting d. Now how about if the client wants to get to e which is mounted in d. Is it legal for the 'smart' client to go back to the ldap server and check for e's location and/or just do a lookup for e in d ? And if it does both and the answer is different from the two sources ?   

We have to distinughish a number of things:  what is decided and written in RFC3530, how servers, clients and administrators might use that to create a limited multi-server namespace (not global but not local either), and what we might someday specify as a global namespace solution for NFS and possibly other protocols.  Note that the last is not very far along.
 
I would say that the current spec leaves it entirely to the clients discretion how it decides what server to initially access and what path to lookup on that server.  If you are assuming that we will have done the work to standardize LDAP (or whatever) for the global namespace function, then I would say that whatever spec  does that would address the question of whether the client might use LDAP directly or somehow find a global root server to get your initial referrals.  The matter is TBD and we all have our own opinions.  So far, in the v4 case, my position is that I haven't seen any good reason, why clients should use direct access to LDAP, since v4 referrals are available.  I haven't thought so much in terms of permitting or prohibiting this.  It is just that I can't see the benefit of it for clients (which makes continual references to clients that do that as 'smart' somewhat grating).
 
As to the a/b/c/d/e-question, the current spec does not explicitly endorse a global namespace, so a/b/c/d/e is a seperate entity and the client can do what it wants.  The question of e in a/b/c/d vs a global a/b/c/d/e doesn't arise.  Until the global namespace is standardized, then this problem doesn't really exist.  Once it is, I assume there have to rules about possible inconsistencies.  If there are multiple referral servers, they may give different answers.  If there are multiple LDAP servers, they may give different answers.  If you have clients accessing both they may give different answers.  Preventing that and telling the client what it should do if it happens is work TBD.
 
What advantage, if any, is there of allowing the smart client to go back to the ldap server to allow it to avoid lookups on the NFS server, if we are to allow this ? 

I don't know.  My feeling is that given that MOVED exists, that it will be used for finding migrated filesystems and, at least in the short-to-medium term for referrals within smaller-scale namespaces, it makes the most sense to expand outward from that core by defining the namespace elements, that we all agree we need.  In that context, my feeling is that direct LDAP access by v4 clients is something to stay away from, unless its utility can be clearly shown, and so far, that hasn't happened.

---

From: Ted Anderson <[email-masked]>
Date: Wed, 07 Jan 2004 15:19:19 -0500

The term "referral" is causing some confusion on at least two levels,
one easily clarified and the other more subtle.

The first confusion arises because NFSV4ERR_MOVED can be returned in two
situations.  In one, a transition occurs (e.g. during LOOKUP) to the
root of a new filesystem[*] which resides on another server.  The other
occurs when most any operation occurs on a file handle and the (whole)
containing filesystem has been migrated (aka relocated, moved) to
another server.  I think the recent exchange between Jon and Dave has
clearly described these two cases.

The cases differ in that the second one uses a file handle obtained from
some previous operation when the filesystem was located on the server.
The cases are related by history in the sense that after a filesystem is
moved a referral is left behind where its root was formerly located in
the parent filesystem.  But what exactly is this referral thingy?

By way of background I'd like to briefly explain how AFS (DFS is very
similar) does this.  In AFS filesystems are called volumes and the
junction between two volumes is called a mountpoint.  An AFS mountpoint
is an actual file system object, implemented as a special symbolic link,
which resides in the parent volume and contains the name of the child
volume.  The AFS client gets the name of the child volume from the
server and then contacts a separate service called the VLDB (Volume
Location Database) to locate the file server hosting this volume.  The
volume name in mountpoint can be qualified by a cell name; a cell is the
AFS term for a file system administrative domain.  The cell name is also
the second component of AFS absolute pathnames.

A key feature of this architecture is the separation of the namespace
construction, via filesystem junctions (AFS mountpoints), and the data
location, handled by a separate service (VLDB).  There are very good
reasons for making this distinction: Nico mentioned some[1] and the
recent discussion about cluster load-balancing highlights another.  The
namespace used by NFSv4 should have this same architectual separation,
even though the implementation might make this much less obvious to
NFSv4 clients than it is to AFS clients.

The NFSv4 client/server protocol does not mention referrals and probably
in many cases servers won't store them in the file system directories
instead they are represented in some other form.  One oft-mentioned
approach imagines a database exported by an LDAP server.  The database
contains two tables, one conceptually listing pathname prefixes and
filesystem names and another table mapping filesystem names to server
locations.  This two level arrangement maintains the architectural
separation described above.  Creating a new filesystem makes an entry in
the second table giving its location, while attaching the filesystem to
the namespace adds an entry to the first table.  Moving a filesystem to
another server modifies only the second table.  The first table, call it
the namespace table, logically encodes the information AFS manages with
its filesystem-resident mountpoints.  The second table, the location
table, is represented in AFS by entries in the VLDB.

The conceptual nature of these two tables and their complete absence
from the NFSv4 protocol is a source of confusion surrounding what we've
been calling referrals.  In the protocol, referrals exist only as
ephemeral file handles whose fs_locations attribute can be queried but
all other operations on them, including GETFH, fail with NFS4ERR_MOVED.
A NFSv4 server that exports several filesystems has them laced together
using pseudo filesystems.  The junctions to real exported filesystems
conceptually represent entries in the namespace table, the location
table entry only notes that the filesystem is local.  When a filesystem
is moved to another server, the location table is updated with the new
host information.  From this perspective, a referral isn't created as a
placeholder for the departed filesystem (as I implied above), the
referral was logically there all along.

We can and probably will continue using the term "referral", but
hopefully this message will provide some details and names for the
underlying concepts.  These may come in handy in further discussions on
the namespace topic.

Ted Anderson

[*] I use "filesystem", one word, to refer a singly-rooted sub-tree
     whose nodes all use the same fsid.  I use "file system" to refer to
     a collection of software and protocols.  A filesystem is mounted
     while a file system is installed.  So you could say "the ext2 file
     system is used to hold the /tmp filesystem".
[1] 
http://www.ietf.org/mail-archive/working-groups/nfsv4/current/msg00589.html

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Carl Burnett <[email-masked]>
Date: Wed, 7 Jan 2004 14:53:34 -0600

Where I think the below model should be different from AFS, is that NFSv4 
clients should consider the NFSv4 file server as their "VLDB". That is 
they should go back to NFS servers to get the location data when they 
receive NFS4ERR_MOVED as part of a lookup sequence that does not yield a 
filehandle. The NFS servers are the constituents of the namespace 
infrastructure (the NFS version of the VLDB). While the AFS/DFS model of 
the clients going somewhere else for the location attributes worked very 
well, it did add complication to the client implementation. It also 
created additional points of failure as well as greater complication for 
administration, deployment, and problem determination. Placing more of 
this in the datacenter behind servers should be an improvement. It also 
facilitates the ideas Jon H. has posted regarding server side traffic 
management.

Carl

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

Ted Anderson <[email-masked]>
Sent by: [email-masked]
01/07/2004 02:19 PM
 
        To:     [email-masked]
        cc: 
        Subject:        Re: [nfsv4] Fw: Global namespace and multiple file 
access protocols

Ted Anderson

---

From: Brent Callaghan <[email-masked]>
Date: Wed, 07 Jan 2004 13:28:05 -0800

> facilitates the ideas Jon H. has posted regarding server side traffic 

If I were a sysadmin, I'd have more confidence in a global namespace
if I could administer it using the same tools as I use for other
nameservice admin functions, in addition to using the same set of
replicated, synchronized, highly-available name servers with
consistent authentication and security.

Whether you force clients to go through the v4 server or not,
chances are that the namespace data is very likely to exist
in some kind of name service that's external to the v4 servers.

Hopefully we can think a bit outside the NFSv4 box and consider
market reality: multi-protocol NAS boxes.  It would be awfully
nice if multi-protocol data access could also apply to a
global filesystem namespace.

	Brent

---

From: Ted Anderson <[email-masked]>
Date: Thu, 08 Jan 2004 09:07:36 -0500

> clients should consider the NFSv4 file server as their "VLDB". That is

I didn't mean to describe the AFS namespace mechanism in order to
suggest that NFSv4 imitate its specifics, but only to provide a concrete
illustration of the architectural separation of naming and location
functions.  One problem with the AFS scheme is the fixed structure of
its pathnames.  Making the second component be the cell name provided
AFS key flexibility in constructing a truly global namespace, but it is
probably too limited for federating filesystems across protocols.  It is
also a concern that the cell name is overloaded to find the location
service.  The namespace system we are considering here needs a more
flexible approach, even if the NFSv4 realization doesn't expose that
flexibility to the clients.

To expand a little further on the reasons for the separation between the
naming and location components of the referral mechanism, I'd like to
point out that the two mappings have very different properties.  The
junction between filesystems is relatively stable and has semantic
meaning.  For example, a pathname entered by a user must always map to
the "same" data (data sameness is a non-trivial issue for another
message).  A change in the namespace table that maps a pathname to
different data is equivalent to a write operation as far as the user is
concerned.  This has implications for access control, server security
and caching.  The contents of the location table are not nearly as
sensitive.  Since clients have a way to determine when they are talking
to an authentic server, they can ensure they are getting the data they
want.  This allows considerable flexibility in collecting, storing, and
distributing location data, as errors and obsolete location information
can be detected, rejected and refreshed when it is used.

But none of this means that in NFSv4 the server wouldn't be a good place
to handle this task to protect clients from some of the complexity.  On
the other hand, a multi-protocol client could contact the namespace
directly, especially when transitioning into or out of NFSv4 space.
Similarly, an NFSv4 server that didn't have namespace support (or
knowledge or access) could use a statically configured response for
fs_locations that always directed the client to a namespace server which
had an NFSv4 front-end.

Ted Anderson