Subject: RE: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Jun 2005 14:58:12 -0400

> I can see that I can get a way with not checking on the server but I
would 
> like to implement a server that is closer to the spec recommendation
and 
> understand the requirements. The "SHOULD" makes me a little
uncomfortable 
> I would prefer "should", but if it is only me I will shut-up.

It sounds like you'd prefer "if you feel like it" and that makes me
uncomfortable.

> I would rather keep using file handles as file ids which can be used
on 
> any node of a cluster file system and not have to encode capabilities
into 
> them. Now the client needs to remember all those different file
handles 
> that should be used for different operations

Whoa!  Before you said you were OK with the client obligation (to only
do
READ/WRITE/etc on filehandles it got from layouts) and only objected
to the work of the server verifying compliance.

Now it appears that you object to forcing the client to obey that rule,
i.e. that the problem with the verification is not that it is hard to 
do but that it would make the client remember "all those different file
handles that should be used for different operations".  If that is the
case then we have a real problem.  You have an implementation in which
every server may act as a metadata server but the pnfs client cannot 
assume that all of the implementations with which it will interact
will have that characteristic or else we have a massive (lack-of)-
interoperability problem.  If a layout tells the client he may use
handle A on server X to READ/WRITE then he had to be capable of 
respecting that, whether the server holds him to it or not.

I'm perfectly OK with exposing additional functionality that a 
cluster fs would provide for metadata load-balancing and failover
as long as we are clear that this is something that the client is
directed to use based on server characteristics.  For example, if
the devinfo entry says that the layout handle may be used to read/
write on a certain set of guaranteed-equivalent servers, then this 
is fine.  Or if a locations_info attribute for the fs indicated that
coherent metadata service was available on a given set of servers,
then this is OK as well.  But each of these options is an option 
and the basic architecture of pnfs is that there is a distinction
between data service and meta-data service and that the client 
has to maintain that distinction.  Just as a pnfs client should 
not use a block address in a SETATTR request or send a filehandle 
in a SCSI block write :-), it should not send a handle it got from 
a layout in a SETATTR request.  It should not send a filehandle it 
got from the meta-data server to a data server unless it has some 
specific guidance that it can, such as a locations_info attribute 
saying servers X, Y, Z are equivalent.  The important point is 
that that latter is not always going to be there and the client
may not assume that it is.    

> and the server verifying that 
> the client did so.

The verification is a big help when testing.  This is going to
be more complicated that what we've done in the past and the 
earlier we detect a problem the better off we are all going 
to be.  I wouldn't think of trying to make this work without 
that kind of checking, particular given all the possible types 
of implementations we have been talking about here.  All this 
requires is one bit in a file handle saying whether it gives the 
right to do all operations (including metadata-server operations) 
or just the subset for data server operations.  If you are inclined 
not to do this, my question would be, "Do you feel lucky?".

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]] 
Sent: Wednesday, June 22, 2005 1:12 PM
To: Noveck, Dave
Cc: Goodson, Garth; [email-masked]; [email-masked]
Subject: RE: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)

"Noveck, Dave" <[email-masked]> wrote on 06/22/2005 06:50:45 AM:

> The full response this time.
> 
> > > You say "you should follow the layout instruction" and I'm torn
> > > between saying "you SHOULD follow the layout instruction" and
> > > "you MUST follow the layout instruction".
> > > 
> > > You say "[should] get an error if you don't" and I say "the server
> > > SHOULD give you an error if you don't".
> > 
> 
> > I say should follow and not MUST follow because I am trying to avoid

the 
> > complication to the server if it MUST enforce this rule which might 
not be 
> > a problem for the server in the first place. 
> 
> Hold on.  I never suggested "MUST" for the server's obligation to
check.
> It seems that the server checking is where you see 
difficulties/inconvenience.
> 
> I did suggest "MUST" as a possiblility for the clients' obligation to 
> conform.  These two do *not* have to go in tandem.  While it wouldn't 
make
> any sense to have the client not have to conform while the server is 
giving
> him an error if he doesn't, it is perfectly reasonable for the spec to
> strongly state the rule for the client but not to insist that the
server
> check for compliance if it has great difficulties doing so.
> 
> > For example, I might want to 
> > allow the read of the same large file(no caching) from one set of
data 

> > server for client A and a different set of data server for client B.

Now 
> > to enforce the above rule the server need to some how encode into
the 
file 
> > handle information about which client can read what from which data 
> > server? 
> 
> I may not be uderstanding your example correctly but it sounds like
the
> case you are worried about is not really at issue here.

The example was just to illustrate the information (capabilities) that
you 
would have to encode in the file handle if we wanted the server verify 
that the client follow the rules.
 
> 
> I know we have been talking kind of loosely about should/SHOULD/MUST 
> "follow the layout instruction".  This is overbroad.  If a server is
> told to use server111.clustersRus.org and takes that same handles and
> uses it on some other server, server111.clustersRus.com for example,
> then he is not following the layout instruction, but the spec is not
> going to require anybody to specifically act to make sure that he 
> gets an error.  The effect of using a filehandle on a server other
> than the one it for had always been undefined, and I expect it will
> continue to be.  Even though your data servers above are going to 
> be in more confederal relationship than the two server111's, I 
> think the same will still hold.  If I take a handle for X and use it
> on Y, I have a real good chance of getting STALE but there is no
> guarantee that I will.
> 
> The specific issue that started this (and that I'm still talking 
> about) is more limited.  I'm given a handle H for a server A in a
> layout and in that is the requirement that that handle be valid
> for READ/WRITE, etc. and not for SETATTR.  If the client uses it
> on A and does a SETATTR, he SHOULD get an error.  If he uses that
> same handle on B and does a READ then he is broken but the server
> has no obligation to recognize handles for other servers.  Similarly
> if he does a SETATTR with the handle on B.
> 
> > I prefer not have this added extra work on the server and just say 
> > that the client should follow the rule to guaranty successful 
operation. 
> 
> In the IETF "should" is very weak and amounts to "gee, it is sort of a
> good idea to".  "SHOULD" is much stronger and says "Do it unless you 
have
> a real good reason not to".  "MUST" just says to do it. 
> 
> I guess I still think that if you receive a handle for server A in a 
> layout, you MUST NOT use it to do operations on that server other than
> PUTFH, COMMIT, READ, WRITE, and that if you do, the server SHOULD give

> you an error.
> 
> If you feel this is too difficult for the server, then the "SHOULD" 
> would give you enough wiggle-room.
> 
I can see that I can get a way with not checking on the server but I
would 
like to implement a server that is closer to the spec recommendation and

understand the requirements. The "SHOULD" makes me a little
uncomfortable 
I would prefer "should", but if it is only me I will shut-up.
I would rather keep using file handles as file ids which can be used on 
any node of a cluster file system and not have to encode capabilities
into 
them. Now the client needs to remember all those different file handles 
that should be used for different operations and the server verifying
that 
the client did so.
Marc. 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Marc Eshel <[email-masked]>
Date: Wed, 22 Jun 2005 13:53:44 -0700

> respecting that, whether the server holds him to it or not.

I don't object. I just voiced a concern about the implementation overhead. 
It is obvious that I am thinking of cluster filesystem only and if there 
is a need for other implementation to require that the client use only the 
file handles provided for each specific operation then fine.
 
> may not assume that it is. 

This sound like a good compromise I would like to see the above options in 
the protocol.
 
> not to do this, my question would be, "Do you feel lucky?".
Like you say it is only one bit and it is not to difficult to implement on 
the server, but now you force the client to remember 2 file handles that a 
different only by one bit (big waste of space :). The client need to 
remember only which are metadata servers and which are data servers and be 
required to direct operation to the appropriate server. If the client made 
a mistake and the server cares than the server can reject the operation.
Marc.

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 22 Jun 2005 20:51:50 -0500

On Wed, Marc Eshel wrote:

<...>

> a mistake and the server cares than the server can reject the operation.

This is a general comment (and one I have made before) based on 
the expectations of what the server will "look" like for a pNFS extension.

It would be prudent identify the various operational, deployment or 
implementation models that people either have planned for the pNFS
functionality or can reasonably imagine.  It will be important so that
we have this in mind when reviewing the general protocol for the
subtle interactions of meta-data and data servers as has been
identified in this thread of discussion.

Oh yeah, the above is with my working group co-chair hat on.

Spencer

---

From: Marc Eshel <[email-masked]>
Date: Wed, 22 Jun 2005 22:11:09 -0700

> Oh yeah, the above is with my working group co-chair hat on.
I just started to think about this topic lately so I don't have a clear 
model so will just dump what I think that I can or would like to do in 
short (I am always very terse but I will try to elaborate:). Give a 
cluster filesystem where all the data is available on all the nodes  I 
would like to use pNFS to do parallel I/O from as many nodes as possible 
so I would make all nodes to be data servers. I believe the metadata 
operation can saturate a single node even if it not doing any data I/O so 
I would like all the nodes to also be metadata server, in other words 
distribute all the operations to all the nodes. Or, direct all clients to 
a specific node for a file or a files segment because it is cached on that 
node or that node has faster access to the disks. Have multiple alternate 
nodes or maybe all the nodes for any given I/O in the case of an error. 
Return the data from the metadata server for small files and avoid all the 
layout exchange and redirection. Have short way to reference a list of 
nodes that can be in the hundreds that can be given once and not repeated 
in every layout. Not have to many requirement to validate correct client 
behavior which requires a lot of book keeping on the server side if the 
only thing it affected is performance (in the case of cluster filesystem) 
after all if the client requested all the data from the metadata server it 
is valid option and no one will produce any error codes. 

I am not sure if this is much help but I plan to spend much more time on 
the topic when I get back from vacation in 3 weeks and provide more input. 
I think that Dave Noveck suggested in his last note to add some options 
that will help with cluster filesystem implementations and I think this is 
a good idea :) maybe we need some more input from other cluster filesystem 
planed or even prototyped implementations.
Marc.

---

From: Brent Welch <[email-masked]>
Date: Thu, 23 Jun 2005 15:25:52 -0700

>>>Marc Eshel said:

I'd like to suggest that we mention the issues about multiple
metadata servers, but that we don't explicitly address them in
the current pNFS proposals.  The goal is to get pNFS clients that
interoperate with different servers.  If some servers have very
different semantics (transparent failover among them, servicing
of metadata or data operations with internal forwarding, whatever)
then that has a big impact on the clients.  In otherwords, we are
starting small with just an effort to distribute the I/O load.
Bypassing the metadata server for I/O goes a long way to reducing
load and providing scalability.  Let's get that worked out before
we do metadata load balancing. 

If you really, really, wanted to go there, then you could define
a new layout type that returned, e.g., a set of equivalent
(deviceID, filehandle) that the client could use based on
the availability or load of the data server.  You might also
be tempted (as Dean is) to return layouts that hint to the client
that if it did a GETATTR to a data server it would get back
something sensible.  However, I don't think we should go there, even
though you and I, as cluster file system implementers may have
already done that.

 > a mistake and the server cares than the server can reject the operation.

I have the same reaction as Dave - I don't see how you can argue that
the spec should imply that the client can get away with switching
around the handles used on the metadata servers and the data servers.
If your implementation wants to give out the same bit pattern for
these cases, that's fine.  But clients simply MUST use the file handles
in the layouts for operations on the corresponding device, and it
is simply undefined what happens if they use a file handle from the
metadata server with a data server or vice versa, or heck, swap around
the file handles among the different data servers.  "Of course" the
clients will keep track of what handles are to be used with what
servers and what operations, because they MUST.

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Brent Welch <[email-masked]>
Date: Thu, 23 Jun 2005 15:50:43 -0700

> planed or even prototyped implementations.

First I'll restate what I think your model is, and then describe another 
one.

Under your cluster file system there is some storage substrate that
today is hidden by your "nodes" (e.g., a back-end SAN).  And, your
nodes cooperate to manage metadata and each exports an identical view.
You are thinking that pNFS will be another layer over your nodes, so
that the underlying storage system is still fairly hidden.  In this model,
pNFS will let you fetch data for a single file from many "nodes" in
parallel, and so get higher bandwidth (ideally) than a single node
can deliver.  Also, by artfully distributing the layouts returned to
clients, you can smear the I/O load over more nodes and achieve more
balanced load among your nodes.  pNFS in its current form does not
directly address the balancing of metadata operations like GETATTR
over your nodes.  The only approach I can offer you is that different
clients mount different nodes to get a coarse level of metadata
load balancing.

As an aside, I think the FS_LOCATIONS attribute is similar in spirit
to what you want, but you want it on a per-file basis.  Today that
operation applies to whole file systems for the purposes of migration.
Ultimately I think we'll want a FILE_LOCATION attribute (or something)
that redirects a client to a different metadata server.  But that would
be a different extension than pNFS.  I think it could be orthogonal.

A different model for your cluster file system is to bring the pNFS
clients more tightly into your cluster file system by exposing more
of the underlying storage layer.  If you had a SAN, for example, then
you'd be giving out block layouts and letting the clients sit right
on the SAN and bypass your nodes altogether to do I/O.

The objects world takes this approach.  The clients can communicate
directly with storage devices, and the storage devices don't really
know how the objects being read/written by clients fit into the
file system.  The clients have to communicate with metadata servers
which take the role of building up file system semantics on top of
something with a simpler interface.  Blocks are really simple, and
objects are slightly richer.  By shunting all the I/O load directly
to the storage devices, the metadata servers don't have all that
much work to do.  So, I'd characterize this as an "asymmetric" model
where data servers own particular pieces of storage, and the metadata
servers direct clients to the appropriate location via layouts.
In contrast, you have a "symmetric" model where any data is available
at any storage server.  But, ultimately there is a hidden asymmetric
model unless you have fully replicated all the data on all nodes
in the symmetric system. 

www.panasas.com
[email-masked]

---

From: Marc Eshel <[email-masked]>
Date: Thu, 23 Jun 2005 17:05:40 -0700

[email-masked] wrote on 06/23/2005 03:25:52 PM:

>  > is a need for other implementation to require that the client use 
only 
>  > This sound like a good compromise I would like to see the above 
options 
> in
Yes I really really want to go there because there are few different 
cluster filesystems out there today with clusters of hounders and 
thousands of nodes and they can really really benefit from the p in pNFS. 
it is not some future requirement and I really don't want to wait for the 
next version of this protocol. I don't want to give hundred identical file 
handles, I want a way to give one file handles and tell the client to use 
it on a list of data servers that I can give once and reference over and 
over. I would also use Dean's hint for GETATTR.

>  > Like you say it is only one bit and it is not to difficult to 
implement 
> on
>  > the server, but now you force the client to remember 2 file handles 
that 
> a
>  > remember only which are metadata servers and which are data servers 
and 
> be
>  > required to direct operation to the appropriate server. If the client 

> made
At first I just suggest that server will not have to verify the usage of 
the right file handle, not that the client swap around file handles (Dave 
said that I don't really have to If I don't want), but like you suggested 
I can use equivalent file handles to avoid the problem, the client have 
nothing to swap or get confused with. Now I suggest like in the above 
comment that the client get only one file handles so there is no 
possibility for confusion and we can save a lot of space.

---

From: Marc Eshel <[email-masked]>
Date: Thu, 23 Jun 2005 17:24:26 -0700

This my last note I am going on vacation and I will not have internet 
access for 3 weeks. I will continue to bug you when I come back.
Marc.

Brent Welch <[email-masked]> wrote on 06/23/2005 03:50:43 PM:

> >>>Marc Eshel said:
>  > 
>  > Spencer Shepler <[email-masked]> wrote on 06/22/2005 
06:51:50 
> PM:

>  > operation can saturate a single node even if it not doing any data 
I/O 
> so

>  > distribute all the operations to all the nodes. Or, direct all 
clients 
> to
>  > a specific node for a file or a files segment because it is cached on 

>  > Return the data from the metadata server for small files and avoid 
all 
>  > after all if the client requested all the data from the metadata 
server 
> it
>  > that will help with cluster filesystem implementations and I think 
this 
> is

> clients mount different nodes to get a coarse level of metadata
I would like to use fs_locations to distribute the client to different 
nodes.

> be a different extension than pNFS.  I think it could be orthogonal.

I don't think that it is orthogonal I was hoping that the output of pNFS 
will include file-locations.

> on the SAN and bypass your nodes altogether to do I/O.

I prefer the file layout protocol on the block one.
 
> model unless you have fully replicated all the data on all nodes
I have a symmetric system the only asymmetric might be in some 
configuration only in regards to performance and I would use the pNFS 
protocol optimize the performance where asymmetry exist. I believe that 
the cluster filesystem model is the simpler one and should be taken into 
consideration in the first version of pNFS.

---

From: "Gordon Waidhofer" <[email-masked]>
Date: Thu, 23 Jun 2005 17:26:45 -0700

I'll plead ignorance right up front because
I haven't had time to read the spec and still don't.
But........

Something (metadata server I guess) is providing a
(deviceID, fileHandle) pair for accessing file content.
How is the deviceID mapped to a node address? Is it possible
that the aggregation of data servers can be handled by the
deviceID->node mapping (here is a list of alternate addresses
for this deviceID)?

I'm not a stake holder in pNFS at this time but think it
likely in the future. I would, for the sake of sound forward
progress, suggest that multiple metadata servers is an order
of magnitude more complicated than the single metadata server
case. It would hopelessly stall pNFS, and useful single metadata
server deployments would stall needlessly because of it. There does
seem to be fair bit of chatter about aggregate devices (clients able
to alternate at will) that it's worth a little try. I agree with
Spencer that case studies would go a long way to helping frame
the mind and the discussion.

FWIW.

Regards,
	-gww

> From: [email-masked] [mailto:[email-masked]]On Behalf Of
> Marc Eshel
> Sent: Thursday, June 23, 2005 5:06 PM
> To: Brent Welch
> Cc: Goodson, Garth; [email-masked]; Noveck, Dave; [email-masked]
> Subject: Re: [nfsv4] pNFS issues/changes (draft-welch-pnfs-ops-02.txt)
>
>