Subject: Re: [nfsv4] Fw: Global namespace and multiple file access protocols

From: Nicolas Williams <[email-masked]>
Date: Fri, 2 Jan 2004 13:28:46 -0600

On Mon, Dec 29, 2003 at 10:35:32AM -0800, Jon Haswell wrote:
> David Robinson wrote:
> > Microsoft's Dfs works in a similar manner, while it does allow the
> > server to indicate a redirection, the client still looks up in AD
> > where to go.
> 
> I mean't to respond to this comment before Christmas but I needed to do a
> little more research before I did. Firstly, after checking with my friends
> in the Samba community, I think the above comment might be misleading -
> depending on what you defined AD as. I believe the following happens in
> CIFS
> 
>  - client tries to access a file or directory
> 
>  - server gives a "path not covered" error. It does not at this stage
>    give any indication as to where the real location is.
> 
>  - client notices the "path not covered" error and sends a TRANS2
>    GetDfsReferral request to the server that gave it the "path not
>    covered" error. The GetDfsReferral request contains the path that
>    the client wanted, relative to that server.
> 
>  - the server responds with a new path (or multiple paths), in full
>    UNC format. The UNC format contains both a server name and a
>    relative path name on that server.
> 
>  - the client connects to that new server and retries the access

Which means that CIFS and NFSv4 referrals are mostly compatible with
each other.  Very nice.

> Now I don't know if there are some special cases that don't follow this
> pattern, but this to me says that CIFS and NFS V4 are very similar in that
> the client goes back to the server returning the error code to find the
> actual data location (in the NFS V4 case at the FS granularity in the CIFS
> case at the directory/file level). How the CIFS file server gets the
> information on where the data now resides I presume would be covered by the
> 'in Active Directory' comment.

Minor nit: the NFSv4 referral granularity is also down to the directory/
file level.

> Looking forward I think it is important that, as somebody else said, we get
> as wide an adoption of the global namespace as possible but also that we do
> not become dependant on every protocol/client supporting it before we move
> forward.
> 
> I would hope to see a model where we have a single global namespace based
> on LDAP or the like. It's client would be NFS V4 servers, CIFS servers and
> possible NFS V3 automounters and the like.

Exactly.  One namespace, referrals for dumb clients, LDAP for smart
servers, and LDAP for smart clients.

> For file serving protocols that support client direction via the server,
> then clients obtain their location information via the native protocols and
> only via their native protocols unless they are given specific permission
> by the servers to do otherwise, i.e. I would see a possible way to support
> this being an additional attribute on an FS which allows a client to access
> the global namespace bypassing the server, if set, for anyfile system
> mountedd within the filesystem to which the attribute is returned.

I'd be a lot less strict myself.

I see no reason why a client can't run the automounter for some things
(e.g., /net, local overrides, to find and mount the global namespace
root(s)) and use referrals for others.

And I see no need for the server to tell the client that the client is
allowed to overlay a local namespace over some filesystem exported by
the server.

Cheers,

Nico
-- 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 2 Jan 2004 13:39:36 -0600

On Wed, Dec 31, 2003 at 10:51:05AM -0800, Jon Haswell wrote:
> On Tuesday, 12/30/2003 at 11:50 PST, "Noveck, Dave"
> <[email-masked]> wrote:

[...]
> > So what's the answer here?  Should we let the automounters drink
> > themselves silly, or is there a way to get them into a treatment
> > program :-)
> >
> 
> So the answer is licensing of the availability of LDAP and the removal of
> organized crime as a result...
> 
> 1) When a client is first booted, regardless of the protocol it supports,
> it may either find the root of the namespace by going to a well known
> server serving using it's protocol or alternatively query LDAP.

1) Client finds its entry point(s) into the namespace with one or more
   of several methods.

> 2) When a client has started interacting with a server and every time it
> crosses into a new FS it must virtually query if locations for FS's mounted
> within this FS should be located via the file serving protocol or
> externally via LDAP.

The server may issue referrals at any time and for any directory or
file, not just at fs cross points.

The client should query for fsid / fs locations voluntarily anyways so
that when it crosses a server-side fs boundary it can find all the
alternative locations of the fs it crossed into.

But NFSv4 referrals need not involve a server-side fs boundary.

> 3) If the protocol the client is using is V2/V3 then the implied result of
> the FS query in 2 is LDAP via automounter

Yes.

> 4) If the protocol the client is using is CIFS then the implied result of
> the query is inband CIFS i.e. TRANS2
> GetDfsReferral to the CIFS server.

Ok.

> 5) If the protocl the client is using is V4 then a real query must made to
> the server and the result returns depends on the implementation of the
> server.

Same as in (4) then (the v4 query is in-band).

> In this way we can have a V3 automounter setup that plugs into a global
> namespace and if the V3 server is a cluster it can load balance via a
> virtualizing switch or whatever. We can also have a V4 server that fits
> into a global namespace but may expose subFS's that it dynamically moves
> between it's nodes and that clients can track without modifying the global
> namespace view in LDAP.
> 
> Everybody happy now ?

I am :)

Cheers,

Nico
--

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 2 Jan 2004 13:14:52 -0800

> I see nothing below that disproves this.  I'm pretty sure that you'd not
> want to limit NFSv4 referrals to server-side fs cross-points, which is
> what I think Jon thinks is the case.

I wouldn't know how to limit NFSv4 referrals to server-side fs cross-points.
As far as the protocol is concerned, the fs boundaries that the clients see
are the only ones that exist.

-----Original Message-----
From: Nicolas Williams [mailto:[email-masked]]
Sent: Friday, January 02, 2004 4:02 PM
To: Noveck, Dave
Cc: Jon Haswell; [email-masked]
Subject: Re: [nfsv4] Fw: Global namespace and multiple file access
protocols

On Fri, Jan 02, 2004 at 12:08:16PM -0800, Noveck, Dave wrote:
> Nico Williams wrote: 
> > [Jon Haswell wrote]
> I don't think so.

[...]
> Minor nit: the NFSv4 referral granularity is also down to the directory/

The statement you object to was in relation to this and should have
read:

"The server may issue referrals at any time and for any directory or
file, not just at _server-side_ fs cross points."

I see nothing below that disproves this.  I'm pretty sure that you'd not
want to limit NFSv4 referrals to server-side fs cross-points, which is
what I think Jon thinks is the case.

On Fri, Jan 02, 2004 at 12:08:16PM -0800, Noveck, Dave wrote:
> Let's suppose that a/b/c is the root of an fs on a given sserver.  So
> suppose you get down to the arbitrary (in terms of fs boudaries) 
> directory a/b/c/d/e and you return the MOVED error.  Depending on
> the fs_root value (the fsid attribute should show the same thing if
> the server chooses to return and the client chooses to fetch it),
> you will have one of two situations: a/b/c/d/e will be the root of 
> a new fs, and thus we have crossed an fs boundary or a/b/c will be
> the root of the current fs in which case we haven't.
> 
> In the first case, we have a referral at an fs boundary.

Nit: the server need not have issued a referral (MOVED), letting the
client detect the server-side fs boundary crossing through the fsid/
fs_locations change.

> In the second case, we still do not have a referral at an arbitrary
> directory.  We have migration of the entire fs rooted at a/b/c
> to some other server.  The way MOVED is defined, it specifies that
> entire fs moves.  In practical terms, this should not be much of
> a problem in that if you want to force a referral at an arbitrary 
> directory, you can, since the server can create his fs boundaries 
> where it wishes, i.e. arbitrarily. 

The third case, which I referred to indirectly is the case where
a/b/c/d/e doesn't exist on the server but the server knows that it does
exist and where, in which case it issues a referral.

Note that in this case a/b/c/d/e is not a server-side fs cross point --
though in a way it is a boundary, in that a/b/c/d/e cannot exist as a
node of the fs that contains a/b/c/d.

Cheers,

Nico
--

---

From: Jon Haswell <[email-masked]>
Date: Fri, 2 Jan 2004 15:50:10 -0800

On Friday, 01/02/2004 at 04:43 CST, Nicolas Williams 
<[email-masked]> wrote:

<snip>

> >
> > 3) If you want to do migration at the directory level in V4 then you 
need
> > to have the server returned a different FSID for each directory. I see 
no
> > sesible way to do this at the file level.
> 
> Yes, fs migration in NFSv4 is fs-level, yes (though I suppose it
> wouldn't be hard to make it more granular), but outside the context of
> fs migration referrals are more granular.

Okay, I'm on vacation so maybe my brain is not functioning but I don't 
understand what mechanisms you are meaning when you say referrals. In my 
mind a client gets the fs_locations attribute when it first enters an FS 
to see if there are more suitable replicas to use and secondly if it gets 
ERR_MOVED it gets fs_locations to find the new locations. Is there some 
other mechanism that is more granular that I have missed ?

> 
> (It's possible to have server-side referral support w/o server-side fs
> migration support -- the protocol uses the same basic facility to
> implement both.)
> 
> [...]
> > By using this approach we can have load balancing with v2/v3 
automounter
> > based on random selection of a server from the LDAP records with 
client to
> > server node affinity lasting until client reboot or similar. With v4
> > however we can have the server actively monitor and load balance 
itself
> > moving clients between nodes on the fly.
> 
> The proponents of smart clients only will say that the v4 server can
> still use the migration facility to load balance smart clients that used
> the directory to get at the fs in question.

But my point here is that the server can't use the migration facility if 
the client is allowed to go off to some other authority, such as an LDAP 
server, to discover locations of data, i.e. if the server returns 
ERR_MOVED to prompt a load balancing event, whcih is not reflected in 
LDAP, then the client must be guaranteed to come back to the server to 
find the new locations data and not go to LDAP.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 2 Jan 2004 18:17:39 -0800

Jon Haswell wrote:
> But my point here is that the server can't use the migration facility if the client 
> is allowed to go off to some other authority, such as an LDAP server, to discover 
> locations of data, i.e. if the server returns ERR_MOVED to prompt a load balancing 
> event, whcih is not reflected in LDAP, then the client must be guaranteed to come 
> back to the server to find the new locations data and not go to LDAP. 

Let's be clear here.  While we are discussing that a possible future spec should say about how a client might
get to the initial location for a file system (direct LDAP access or referrals from v4 servers), the question of
how you deal with a MOVED error is clear in RFC3530.  The client is supposed to get fs_locations from the
server and use that to find the new location of the file system.  If it does something else, it is violating the
spec.
 
It is in the clients' interest to the use server-provided fs_locations value (and not just because Jon, aka Elliott N,
would bust them otherwise :-).  The server may have transferred the existing state (open files and locks)  to the
new server on behalf of the client so that the client, by using the server's instructions may take advantage of this.
If client decided to ignore fs_locations and get location information from LDAP (because it is a "smart" client) it 
would be too smart for its own good.

-----Original Message-----
From: Jon Haswell [mailto:[email-masked]]
Sent: Friday, January 02, 2004 6:50 PM
To: Nicolas Williams
Cc: Jon Haswell; [email-masked]
Subject: Re: [nfsv4] Fw: Global namespace and multiple file access protocols

On Friday, 01/02/2004 at 04:43 CST, Nicolas Williams <[email-masked]> wrote: 

<snip> 

> >
> > 3) If you want to do migration at the directory level in V4 then you need
> > to have the server returned a different FSID for each directory. I see no

Okay, I'm on vacation so maybe my brain is not functioning but I don't understand what mechanisms you are meaning when you say referrals. In my mind a client gets the fs_locations attribute when it first enters an FS to see if there are more suitable replicas to use and secondly if it gets ERR_MOVED it gets fs_locations to find the new locations. Is there some other mechanism that is more granular that I have missed ? 

> > By using this approach we can have load balancing with v2/v3 automounter
> > based on random selection of a server from the LDAP records with client to
> > however we can have the server actively monitor and load balance itself

But my point here is that the server can't use the migration facility if the client is allowed to go off to some other authority, such as an LDAP server, to discover locations of data, i.e. if the server returns ERR_MOVED to prompt a load balancing event, whcih is not reflected in LDAP, then the client must be guaranteed to come back to the server to find the new locations data and not go to LDAP.

---

From: Jon Haswell <[email-masked]>
Date: Sat, 3 Jan 2004 22:27:27 -0800

On Friday, 01/02/2004 at 06:17 PST, "Noveck, Dave" 
<[email-masked]> wrote:

<snip>

> get to  the initial location for a file system (direct LDAP access or 
referrals 
> from v4  servers), the question of
> how  you deal with a MOVED error is clear in RFC3530.  The client is 
supposed 
> to  get fs_locations from the
> server  and use that to find the new location of the file system.  If it 
does 
> something else, it is violating the
> spec.

Okay it's good to have you state this as I had not been assuming this. I 
assume we would apply this rule to any protocol capable of issuing 
ERR_MOVED, i.e. CIFS and NFS V4. 

Now how do I extrapolate this to walking down a tree in the global 
namespace. If we walking down this global namespace with a 'dumb' client 
aren't we getting ERR_MOVED whenever we cross a server boundary, i.e. if 
we are walking down a/b/c/d and a/b/c are virtual on a root server and d 
is a real FS on another server we will get ERR_MOVED when we lookup d - 
correct ? Hence it would be illegal for any client to do anything in this 
situation except get the fs_locations from the server - correct ?

Now we're saying a smart client may walk down a/b/c/d on the ldap server 
and then directly access the NFS server exporting d. Now how about if the 
client wants to get to e which is mounted in d. Is it legal for the 
'smart' client to go back to the ldap server and check for e's location 
and/or just do a lookup for e in d ? And if it does both and the answer is 
different from the two sources ? 

What advantage, if any, is there of allowing the smart client to go back 
to the ldap server to allow it to avoid lookups on the NFS server, if we 
are to allow this ? 

>  
> It is  in the clients' interest to the use server-provided fs_locations 
value 
> (and not  just because Jon, aka Elliott N,
> would  bust them otherwise :-).  The server may have transferred the 
existing 
> state (open files and locks)   to the
> new  server on behalf of the client so that the client, by using the 
> server's instructions may take advantage of this.
> If  client decided to ignore fs_locations and get location information 
from 
> LDAP  (because it is a "smart" client) it 

Thanks

Jon Haswell
IBM Research

---

From: Jon Haswell <[email-masked]>
Date: Mon, 5 Jan 2004 10:47:26 -0800

On Sunday, 01/04/2004 at 04:31 PST, "Noveck, Dave" <[email-masked]>
wrote:
<snip>

>> assume we would apply this rule to any protocol capable of  issuing
ERR_MOVED,
>> i.e. CIFS and NFS V4.
>
> I'm only capable  of talking about v4.  I don't know anything about what
rules
> CIFS  has.  As I understand the process, Microsoft builds clients, they
do
> what  they do, and servers adapt, making the promulgation of any rules we
might
> decide  on rather irrelevant.

Agreed, we can't mandate what Microsoft will do, however we can influence
them and also the CIFSFS and Samba code.

<snip>

>
>> Now we're saying a smart  client may walk down a/b/c/d on the ldap
server and
>> then directly access the NFS  server exporting d. Now how about if the
client
>> wants to get to e which is  mounted in d. Is it legal for the 'smart'
client to
>> go back to the ldap server  and check for e's location and/or just do a
lookup
>> for e in d ? And if it does  both and the answer is different from the
two
>> sources  ?
>
> We have to distinughish a number of  things:  what is decided and written
in
> RFC3530, how servers, clients  and administrators might use that to
create a
> limited multi-server namespace  (not global but not local either), and
what we
> might someday specify as a global  namespace solution for NFS and
possibly
> other protocols.  Note that the  last is not very far along.
>
> I  would say that the current spec leaves it entirely to the clients
> discretion how it decides what server to initially access and what path
to
> lookup on that server.  If you are assuming that we will have done the
work  to
> standardize LDAP (or whatever) for the global namespace function, then I
would
> say that whatever spec  does that would address the  question of whether
the
> client might use LDAP directly or somehow  find a global root server to
get
> your initial referrals.  The matter is TBD  and we all have our own
opinions.
> So far, in the v4 case, my position is  that I haven't seen any good
reason,
> why clients should use direct access to  LDAP, since v4 referrals are
> available.  I haven't thought so much in terms  of permitting or
prohibiting
> this.  It is just that I can't see the benefit  of it for clients (which
makes
> continual references to clients that do that as  'smart' somewhat
grating).

I absolutely agree with you - and isn't the point of this discussion being
to try and form a consensus of what to put into the future spec that
standardizes this around LDAP etc

>
> As to  the a/b/c/d/e-question, the current spec does not explicitly
endorse a
> global  namespace, so a/b/c/d/e is a seperate entity and the client can
do what
> it  wants.  The question of e in a/b/c/d vs a global a/b/c/d/e doesn't
arise.
> Until the global namespace is standardized, then this problem  doesn't
really
> exist.  Once it is, I assume there have to rules about  possible
> inconsistencies.  If there are multiple referral servers, they may  give
> different answers.  If there are multiple LDAP servers, they may give
> different answers.  If you have clients accessing both they may give
different
> answers.  Preventing that and telling the client what it should  do if it

> happens is work TBD.
to
> the ldap  server to allow it to avoid lookups on the NFS server, if we
are to
> allow this  ?
>
> I don't know.  My feeling is that given that MOVED  exists, that it will
be
> used for finding migrated filesystems and, at  least in the
short-to-medium
> term for referrals within smaller-scale namespaces,  it makes the most
sense to
> expand outward from that core by defining the  namespace elements, that
we all
> agree we need.  In that context, my feeling  is that direct LDAP access
by v4
> clients is something to stay away from, unless  its utility can be
clearly
> shown, and so far, that hasn't  happened.
>
I agree

---

From: Jon Haswell <[email-masked]>
Date: Thu, 8 Jan 2004 10:41:51 -0800

On Wednesday, 01/07/2004 at 01:28 PST, Brent Callaghan <[email-masked]>
wrote:

> Whether you force clients to go through the v4 server or not,
> chances are that the namespace data is very likely to exist
> in some kind of name service that's external to the v4 servers.
>
> Hopefully we can think a bit outside the NFSv4 box and consider
> market reality: multi-protocol NAS boxes.  It would be awfully
> nice if multi-protocol data access could also apply to a
> global filesystem namespace.
>

I agree but I don't think anybody here is in disagreement. We all, and I
believe this includes Carl, want the global namespace to reside in some
external existing database, i.e. LDAP or something similar. All Carl, Dave
and I are saying is that a V4 client should not directly access this when
it gets ERR_MOVED or the like. Probably nor should CIFS. However V3
automounter could as that fits better into the automounter view of the
world.

Jon Haswell
IBM Research