Subject: RE: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Jun 2005 09:50:45 -0400

The full response this time.

> > You say "you should follow the layout instruction" and I'm torn
> > between saying "you SHOULD follow the layout instruction" and
> > "you MUST follow the layout instruction".
> > 
> > You say "[should] get an error if you don't" and I say "the server
> > SHOULD give you an error if you don't".
> 

> I say should follow and not MUST follow because I am trying to avoid the 
> complication to the server if it MUST enforce this rule which might not be 
> a problem for the server in the first place. 

Hold on.  I never suggested "MUST" for the server's obligation to check.
It seems that the server checking is where you see difficulties/inconvenience.

I did suggest "MUST" as a possiblility for the clients' obligation to 
conform.  These two do *not* have to go in tandem.  While it wouldn't make
any sense to have the client not have to conform while the server is giving
him an error if he doesn't, it is perfectly reasonable for the spec to
strongly state the rule for the client but not to insist that the server
check for compliance if it has great difficulties doing so.

> For example, I might want to 
> allow the read of the same large file(no caching) from one set of data 
> server for client A and a different set of data server for client B. Now 
> to enforce the above rule the server need to some how encode into the file 
> handle information about which client can read what from which data 
> server? 

I may not be uderstanding your example correctly but it sounds like the
case you are worried about is not really at issue here.

I know we have been talking kind of loosely about should/SHOULD/MUST 
"follow the layout instruction".  This is overbroad.  If a server is
told to use server111.clustersRus.org and takes that same handles and
uses it on some other server, server111.clustersRus.com for example,
then he is not following the layout instruction, but the spec is not
going to require anybody to specifically act to make sure that he 
gets an error.  The effect of using a filehandle on a server other
than the one it for had always been undefined, and I expect it will
continue to be.  Even though your data servers above are going to 
be in more confederal relationship than the two server111's, I 
think the same will still hold.  If I take a handle for X and use it
on Y, I have a real good chance of getting STALE but there is no
guarantee that I will.

The specific issue that started this (and that I'm still talking 
about) is more limited.  I'm given a handle H for a server A in a
layout and in that is the requirement that that handle be valid
for READ/WRITE, etc. and not for SETATTR.  If the client uses it
on A and does a SETATTR, he SHOULD get an error.  If he uses that
same handle on B and does a READ then he is broken but the server
has no obligation to recognize handles for other servers.  Similarly
if he does a SETATTR with the handle on B.

> I prefer not have this added extra work on the server and just say 
> that the client should follow the rule to guaranty successful operation. 

In the IETF "should" is very weak and amounts to "gee, it is sort of a
good idea to".  "SHOULD" is much stronger and says "Do it unless you have
a real good reason not to".  "MUST" just says to do it. 

I guess I still think that if you receive a handle for server A in a 
layout, you MUST NOT use it to do operations on that server other than
PUTFH, COMMIT, READ, WRITE, and that if you do, the server SHOULD give 
you an error.

If you feel this is too difficult for the server, then the "SHOULD" 
would give you enough wiggle-room.

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]]
Sent: Tuesday, June 21, 2005 6:27 PM
To: Noveck, Dave
Cc: Dean Hildebrand; Goodson, Garth; [email-masked];
[email-masked]
Subject: RE: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

"Noveck, Dave" <[email-masked]> wrote on 06/21/2005 01:51:07 PM:

> > With fs-locations we give the client a list of metadata server 
> > that can act as the server for given filesystem and the client 
> > can choose which one to use and switch among them at will or 
> > because of some failure. 
> 
> The problem is that in v4.0 as it stands there is an enormous
> range of ways that a client can interpret the fs_locations list:
> 
> "Here is a list of servers and you can switch when there is failure,
> after refetching your change attibutes (because change attribute
> is within the purview of a particular server)."  (and that is not the
> most extreme case of discontinuity -- some people are thinking that 
> filehandles and fileids will just change or that you may wind up
> with a slighly out-of-data version of the data).
> 
> "Here is a list of servers and you can switch when there is a 
> failure with no discontinuity of access (changes in fh's, stateid's,
> fileids, change attributes), although changing is a big deal
> and you shouldn't do it without good cause."
> 
> "Here is a list of servers and you can switch when there is a
> failure with no discontinuity of access or even at will since
> there is no big cost to switch."
> 
> "Here is a list of servers and you can access any of these 
> servers as you will at the same time (multi-pathing or a 
> cluster fs) since they are all effectively the same thing".
> 
> The problem is that the client only knows about the list and
> has no way of knowing which of the statements above is associated
> with the list of servers he is getting.  I have been thinking
> about a locations_info attribute for 4.1 that would allow the
> server to tell the client which of those he meant and also 
> give preference information (local vs. remote copies, absolutely
> up-to-data vs. slightly out-of-data copies).
> 
> > is solely within the purview of a particular server
> 
> > I agree that once you got a layout for a file you should follow 
> > the layout instruction and read/write only from the specified 
> > nodes and get an error if you don't. 
> 
> Then it sounds like we are in violent agreement, except maybe for
> choice of modal auxiliaries or capitalization.
> 
> You say "you should follow the layout instruction" and I'm torn
> between saying "you SHOULD follow the layout instruction" and
> "you MUST follow the layout instruction".
> 
> You say "[should] get an error if you don't" and I say "the server
> SHOULD give you an error if you don't".
> 

I say should follow and not MUST follow because I am trying to avoid the 
complication to the server if it MUST enforce this rule which might not be 
a problem for the server in the first place. For example, I might want to 
allow the read of the same large file(no caching) from one set of data 
server for client A and a different set of data server for client B. Now 
to enforce the above rule the server need to some how encode into the file 
handle information about which client can read what from which data 
server? I prefer not have this added extra work on the server and just say 
that the client should follow the rule to guaranty successful operation. 
Marc. 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Dean Hildebrand <[email-masked]>
Date: Wed, 22 Jun 2005 11:23:58 -0400 (EDT)

Two comments:

1) I think another example to keep in mind is that a layout could be
redundant.  You can imagine layouts that just return "file available on
all data servers", and then let the client use its own load balancing
scheme to access data.  Of course this depends on all clients using the
same load balancing algorithm, but file systems already exist in this
manner.

2) I think it can be safely assumed that someone is going to want to use
these extensions to send GETATTR, etc to the data servers to offload work
from the metadata server.  Garth's NASD paper did this with NASD NFS and
showed the benefits.  If one layout driver redirects GETATTR's, etc and
another one doesn't, I assume the faster one will be used, not the one
that follows the spec.  If we would rather put this ability into a new
FILE_LOCATIONS attribute or something similar, then we should say so.  I
wrote a FILE_LOCATIONS internet draft about 2 years ago that never went
anywhere....

Dean

---

From: Marc Eshel <[email-masked]>
Date: Wed, 22 Jun 2005 10:12:26 -0700

> case you are worried about is not really at issue here.

The example was just to illustrate the information (capabilities) that you 
would have to encode in the file handle if we wanted the server verify 
that the client follow the rules.
 
> would give you enough wiggle-room.
I can see that I can get a way with not checking on the server but I would 
like to implement a server that is closer to the spec recommendation and 
understand the requirements. The "SHOULD" makes me a little uncomfortable 
I would prefer "should", but if it is only me I will shut-up.
I would rather keep using file handles as file ids which can be used on 
any node of a cluster file system and not have to encode capabilities into 
them. Now the client needs to remember all those different file handles 
that should be used for different operations and the server verifying that 
the client did so.
Marc.