Subject: Re: [nfsv4] pNFS and equivalent data servers

From: Marc Eshel <[email-masked]>
Date: Thu, 1 Sep 2005 00:01:40 -0700

Brent Welch <[email-masked]> wrote on 08/31/2005 03:39:28 PM:

> I'm worried we are conflating three different things:
> multiple paths to the same device,
> redundant or "equivalent" data servers,
> error recovery.
> 
I don't see them as different thing at all, they all have a related 
objective which is high availability. 

> In addition, we are trying to optimize performance of an error case,
> which should be viewed with suspicion. The important thing
> to get right in the failure case is that storage is consistent,
> and that the server implementation isn't horribly complex,
> and that the optimization for the error case doesn't slow
> down the non-error case.
> 
Suspicion that it is used for other than error recovery case? what else 
can you used it for? if you have some good ideas there please share:) I 
don't see how this slows down non-error case. In an setup where you have 
few hundred nodes and few thousands disks and you stripe mutli gigabyte 
file over all those parts, error conditions are routine and should not be 
regarded as isolated events.

> Finally, you are also making some assumptions about the back
> end that don't apply everywhere.  For example, the analog of
> a file handle in the object world is a capability, and they are
> by definition specific to a device.  You simply can't give them
> to the wrong device and have them work.  I can also imagine
> there could be files-based pNFS back-ends where the servers don't
> understand each others FH.
> 
This an option that you can use if it help your back-ends and ignore 
otherwise. I am not sure about objects but I can see a block data server 
that can make use of this option.

> Therefore, I think:
> 
> If you have multiple paths to exactly the same device, then
> it is appropriate to put that information into the address(s)
> returned with GETDEVINFO, and I'd be fine with that in the spec.
> I'll note that this can also be viewed as a routing issue, and
> you can have multiple paths (i.e., multiple routers) to get
> between two IP addresses without needing multiple IPs at the
> end points.  (In otherwords, multipath routing can be a matter
> confined entirely to the networking configuration.)
> 
> However, if you are layering pNFS over a cluster file system
> (remember, one goal is to not go through a file system, but
> go to the storage devices they use) and it so happens that
> different "devices" understand the same file handles, then
> that is a private matter for your layout type.  It just isn't
> generic enough to impose on everybody.  If you do anything,
> I think it must be 2(b), and it is a new files layout type where
> there are multiple, distinct devices that are "equivalent"
> in Dave Noveck's terms. 

2(b) can work too, but I don't see how you consider it an imposition if 
the metadata server can just have one address per device id and ignore the 
option of having more addresses.
 
> 
> Let's go back to the error handling case because it is important.
> Our strategy is that clients notify the server when they encounter
> errors.  Note that in a wacked out customer network configuration,
> the server-to-server visibility can be different than the
> client-to-server communication paths. (I.e., your heartbeat
> scheme may not detect the same problems your clients see.)
> In response, our servers make their own judgment about the health
> of the devices.  They do extra work to verify that the file is
> in good shape.  They may in fact choose to give the client a
> different layout that reflects whatever error recovery has
> occurred (or is in progress).  Or, they may conclude that storage
> is fine and the client is just confused, and so not recall any
> layouts. This may sound long winded or overly general.
> But, my assertion is that you cannot be too
> careful, and any "optimizations" you are tempted to do in the
> error case will result in corrupt data at some point.  Or, it
> means that the metadata server has to revalidate storage all
> the time, which will make normal-case performance slow.  I've
> explored both alternatives :-/
 
What is the protocol for the client to notify the server of an error?

> Hmm - I probably went off a bit on the error recovery.  A simpler
> way to state this is:
> 
> 1. GETDEVICEINFO is about associating an address (or addresses) with
> a storage device.  Overloading the semantics of what it means
> to have multiple addresses is not appropriate.  All it means is
> that if you send a message to any of the addresses it gets to
> the exact same device.
> 
> 2. Layouts that support redundancy are cool and useful.  I'm all
> for including them in the spec as different kinds of layouts.  However,
> error handling strategies for these schemes are likely to be
> very specific to the semantics of that particular layout, and
> trying to apply those semantics to all layouts is unlikely to work.
> 
> >>>Marc Eshel said:
>  > 
>  > I prefer option 2a too. Changing getdevicelist and getdeviceinfo to 
>  > optionally return multiple dev_addrs per dev_id is a good idea for 
error 
>  > recovery.
>  > Marc. 
>  > 
>  > [email-masked] wrote on 08/31/2005 11:44:53 AM:
>  > 
>  > > There has been some private email floating around about the desire 
for 
>  > > pNFS to enable the client to use alternative data servers to serve 
the 
>  > > same data stripe.
>  > > 
>  > > This can be useful in the case of a data server failure.  In this 
case 
>  > > the metadata server need not recall all layouts identifying the 
failed 
>  > > data server.  Instead, clients can automatically switch to using a 
>  > > different data server.
>  > > 
>  > > I believe that there are three main options.  I believe consensus 
is 
>  > > concentrating around option 2.  Feedback is welcome.
>  > > 
>  > > 1) Don't change the spec; leave it to the metadata server to recall 

>  > > layouts or give layouts with the appropriate IDs for load 
balancing, 
>  > > etc.  Failure cases would require the client to write through the 
>  > > metadata server, or possibly return the layout, request a new one 
and 
>  > > depend on the metadata server handing out layout with a set of 
> different
>  > 
>  > > devices (e.g., depend on the metadata server to remap the layout).
>  > > 
>  > > 2) Change the pNFS spec to: (a) include a mapping of either one 
device 
>  > > ID to multiple dev_addrs or (b) provide a list of device IDs in the 

>  > > layout.  Either option would include the restriction that the 
devices 
>  > > identified are equivalent (i.e., can handle the same FH, stateid, 
>  > > logically share the same storage, etc).
>  > > 
>  > > Dave Noveck and I prefer 2a, while others prefer 2b.  I prefer 2a 
>  > > because it simplifies the layouts, can be used by other layout 
types, 
>  > > and can be handled by a lower layer (e.g., at the device level 
below 
> the
>  > 
>  > > layout level).
>  > > 
>  > > Dave Noveck's comments on (2):
>  > > Requiring that these devices be equivalent would mean, I think, 
that
>  > > the basic purpose of this essentially multi-pathing and that in the
>  > > event of an IO error it doesn't provide additional capability.  If
>  > > the two data servers have sufficient connection that writing to one
>  > > is equivalent to writing to the other if there are multiple
>  > > copies, reading from one (if it is the only one the MDS gave you)
>  > > should give you access to all of the copies that there are.  In the
>  > > case of data server failure though, I can see that having a large
>  > > number of clients with a large number of layouts trade them in
>  > > might be troublesome for the MDS.  So having people switch on their
>  > > own would be a help.
>  > > 
>  > > 3) Look at it from a coarser granularity and try to use/extend the 
>  > > location_info attribute applied to a data server.  This would 
require 
>  > > being able to request a list of 'equivalent' data servers, probably 

> from
>  > 
>  > > the metadata server, which is different than the use for which it 
was 
>  > > initially designed.
>  > > 
>  > > This seems to be the least attractive option partly due to the 
> increased
>  > 
>  > > complexity.  As well, it does not seem to be the best fit.
>  > > 
>  > > -Garth
>  > > 
>  > > _______________________________________________
>  > > nfsv4 mailing list
>  > > [email-masked]
>  > > https://www1.ietf.org/mailman/listinfo/nfsv4
>  > 
>  > 
>  > _______________________________________________
>  > nfsv4 mailing list
>  > [email-masked]
>  > https://www1.ietf.org/mailman/listinfo/nfsv4
>  > 
>  > 
> 
> --
> Brent Welch
> Software Architect, Panasas Inc
> Accelerating Time to Results(tm) with Clustered Storage
> 
> www.panasas.com
> [email-masked]
> 
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Thu, 01 Sep 2005 09:41:29 -0400

brent, what do you think of this.

i want to export your panasas fs to another proprietary cluster fs across the 
WAN. so, i place pNFS meta/data servers on panasas client nodes, and export 
the panasas fs to the WAN. i do the same for, say, lustre, GPFS, or a block 
layout cluster. i can then use the generic file layout clients with full 
nfsv4.0 security to access both clusters and copy your panasas data from one 
cluster to another. in this scenerio, the pNFS servers exporting the panasas 
cluster are in the same situation as pNFS servers on a GPFS cluster, and have 
the same issues - each pNFS server can be a metadata server or data server, 
multiple paths, etc. this is a good use of pNFS, and should be considered in 
the protocol design.

-->Andy 

>  > optionally return multiple dev_addrs per dev_id is a good idea for error 
>  > recovery.
>  > Marc. 
>  > 
>  > > There has been some private email floating around about the desire for 
>  > > pNFS to enable the client to use alternative data servers to serve the 
>  > > same data stripe.
>  > > 
>  > > This can be useful in the case of a data server failure.  In this case 
>  > > the metadata server need not recall all layouts identifying the failed 
>  > > I believe that there are three main options.  I believe consensus is 
>  > > layouts or give layouts with the appropriate IDs for load balancing, 
>  > > metadata server, or possibly return the layout, request a new one and 
>  > > 2) Change the pNFS spec to: (a) include a mapping of either one device 
>  > > layout.  Either option would include the restriction that the devices 
>  > > because it simplifies the layouts, can be used by other layout types, 
>  > > and can be handled by a lower layer (e.g., at the device level below 
> the
>  > 
>  > > layout level).
>  > > 
>  > > Requiring that these devices be equivalent would mean, I think, that
>  > > location_info attribute applied to a data server.  This would require 
>  > > the metadata server, which is different than the use for which it was 
>  > > initially designed.
>  > >

---

From: Brent Welch <[email-masked]>
Date: Thu, 01 Sep 2005 15:49:44 -0700

Sure, Andy, what you suggest is fine.  It doesn't relate directly
to the debate (as I understand it), which is about the semantics of
having multiple addresses for one "device".

I agree when Marc says that multi-path routing, redundant servers,
and robust error handling are all related.  But, I'm only trying
to point out that they are not the same thing;  instead, they are
different parts of an overall architecture.  I am worried that
folks want to burn into the pNFS spec that if a device has multiple
addresses, it might actually be a magic device that lives in two
physical locations and has some semantics about the data in those
two locations.  That is not a "device" -> that is a "layout".
That's exactly what a layout is, right? it is a distribution of data
across multiple devices, with various semantics about the relationship
between the distributed data.

As I said, I am completely in favor of allowing a single device to
have multiple addresses to support multi-path access to that device.
We have to agree that a device is a block storage device, or an
object storage device, or a file server.  In contrast, an NFS
head over top a cluster file system is not a "device".  It is something
much more clever with loads of semantics, and hiding those semantics
in GETDEVICEINFO is not the right thing to do.

Next, my comment about optimization of the error case is this:
Be careful that your optimizations don't actually make error recovery
more difficult, and ultimately lead to complex servers, data corruption
bugs, or both.  Error recovery is a property of the layout
(i.e., it is a property of the relationship among the distributed
data)  If you want to define a layout that has fast and loose
"optimized" error handling, that's fine.  I'm not sure I'll choose
to use it until I understand how it works.  And, I don't want 
assumptions based on those kind of layouts to bleed into the generic
layers of pNFS and screw me over.  That's all.

>>>"William A.(Andy) Adamson" said:
 > 
 > i want to export your panasas fs to another proprietary cluster fs 
across th
     e 

 > cluster are in the same situation as pNFS servers on a GPFS cluster, and 
hav
     e 
      
 > >  > optionally return multiple dev_addrs per dev_id is a good idea for 
erro
     r 
 > >  > > There has been some private email floating around about the 
desire fo
     r 
 > >  > > pNFS to enable the client to use alternative data servers to 
serve th
     e 
 > >  > > This can be useful in the case of a data server failure.  In this 
cas
     e 
 > >  > > the metadata server need not recall all layouts identifying the 
faile
     d 
is
balancing,
and
      
 > >  > > 2) Change the pNFS spec to: (a) include a mapping of either one 
devic
     e 
devices
      
types,
      
below
that
require
      
was

www.panasas.com
[email-masked]

---

From: Marc Eshel <[email-masked]>
Date: Thu, 1 Sep 2005 19:19:01 -0700

> between the distributed data.

The main motivation here is to get multi-path to the same device. When one 
of the node in the cluster fails (for example, the IO request times out) 
the client can use a different node in the cluster to get to the same 
devices.
 
> in GETDEVICEINFO is not the right thing to do.

It is fine with me if the alternate devices are specified in the layout 
(option 2b). Yes cluster file system have some advantages but I am not 
trying to hide any semantics. 
 
> layers of pNFS and screw me over.  That's all.
 
If you think this option will make a server to complex don't use it. On 
the client side the different implementations can chose to use the 
alternate device on error or ignore it with no impact on the default 
behavior. A client that chooses to use it will just have to test for 
alternate device and retry the operation with the alternate device. Simple 
enough, and if it was talking to a server that doesn't not have alternate 
devices the recovery path is not impacted at all. 
I would say that adding a different file layout for each trivial option as 
this one would increase the complexity on both client and server that 
would have to deal with yet another layout. 
Compare that to what is proposed for locations_info where all possible 
attribute are defined and different implementations can use the sub-set 
that would be useful for them.
Marc.

---

From: Marc Eshel <[email-masked]>
Date: Fri, 2 Sep 2005 10:39:51 -0700

Since a new draft is coming out soon I wanted to bring up few changes. I 
think that LAYOUTGET should have the same capability that GETDEVICELIST 
has to get a long list of devices in multiple calls using a cookie. 
Another request that I made before is to be able to recall the device list 
from the client (invalidate its device list cache) so the server doesn't 
have to maintain this for ever increasing device id. Garth said 
"reasonable request", Dave Noveck said "no", does anyone else has an 
opinion? This would be my first choice but if there is no consensus for it 
I would at least request that the device id change from 32bit to 64 bit.

---

From: Garth Goodson <[email-masked]>
Date: Fri, 02 Sep 2005 11:01:28 -0700

> has to get a long list of devices in multiple calls using a cookie.

The reason we haven't done this is since we are using a shorthand for 
the device ID (8 bytes -- I've increased it to 64 bit) and since the 
server can always hand back a layout that covers a smaller byte range 
than was asked for (in the case of a very long list).  The client can 
always make calls to get multiple layouts to cover the range it needs 
(if it really can't fit in the 32KB or whatever the transport allows). 
We could add a field that specifies the maximum byte count to ensure the 
appropriate size is returned.

> opinion? 

I did say that, but I'm still not convinced it is necessary.  I think 
the device ID space is large enough that reuse of IDs is not an issue. 
The client is able to maintain its own cache of IDs, since it can always 
reget them.  If mappings change, the server can recall the appropriate 
layouts and clients can demand the new IDs.  I understand that you want 
to avoid another level of ID/node mapping, but depending on the 
multi-pathing issue, you would have to do so anyway.

> This would be my first choice but if there is no consensus for it 
This is fine, and a change I've just made.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Fri, 2 Sep 2005 11:02:13 -0700

The last draft say in 2.4 "Storage Protocol" that the data server can 
NFSv3. I assume that would be a new layout type, can we add it to the 
draft?
Marc.

---

From: Garth Goodson <[email-masked]>
Date: Fri, 02 Sep 2005 11:13:08 -0700

> NFSv3. I assume that would be a new layout type, can we add it to the 

It is saying that v3 is a possible storage protocol for files (perhaps I 
should remove this reference, or clarify it).  We have avoided defining 
such a storage protocol since it doesn't fit within the v4 security 
framework.  It could be added later as an additional protocol (such as 
is being done for blocks and objects), however, the main spec is only 
going to specify v4 as a storage protocol.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Fri, 2 Sep 2005 11:48:57 -0700

> appropriate size is returned.

If you are describing the striping layout where you can fit only 9 out of 
your 10 devices in the first message that means that now you will need to 
get the rest of the layout for the file in a one to one mapping which can 
be a large number of messages for a large file, but if you could say that 
this is a continuation of the first messages than you can get the complete 
set in 2 messages. Having said that I would say that if we can send a 
message of size 32K which can describe about 4000 device that should be 
enough, but less would be a problem. 
Marc.

---

From: Marc Eshel <[email-masked]>
Date: Fri, 2 Sep 2005 15:40:08 -0700

I am trying to figure out if there are some conditions under which we 
don't need the extra messages, in the file protocol, from the metadata 
server to all the data servers using  the back-end management protocol for 
each open/close or other state changes.  A good solution would be if we 
can package the capabilities and let the client present it to the data 
server like the suggestion in the "[nfsv4] pNFS, OPEN principals and ACLs" 
thread. But if we don't have such a solution, would it be possible, say 
for a read only exported file system, to use the special stateid of all 
bits 0 or 1 for I/O to data servers? If the answer is yes, then how does 
the metadata server tell the client to use the special stateid? I assume 
that we still want to use valid stateid for communicating with the 
metadata server.

Marc.

---

From: Brent Welch <[email-masked]>
Date: Wed, 07 Sep 2005 09:22:16 -0700

Right - there generally isn't a problem with big layouts in
the files or objects case.  In the blocks world a layout might
contain many block (or extent) addresses, and the layout can 
quickly get large.  The model is that the server bounds the
size of the layout data structure, and this covers an explicit
range of the file's contents.  We don't need the complexity of
a readdir cookie model - even the existing block-based SAN file systems
don't take that approach.  

 > > > Since a new draft is coming out soon I wanted to bring up few 
changes.
 > I 

www.panasas.com
[email-masked]

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 7 Sep 2005 12:55:37 -0400

On Wed, Sep 07, 2005 at 09:22:16AM -0700, Brent Welch wrote:
> the files or objects case.

Do the current proposals have an explicit bound on the size?  If not,
it'd be nice to choose one.  It might simplify client implementation,
but mainly it'd be a useful exercise just to make sure we agree what's
too "big"....

I think the layout data for a single stripe is about 200 bytes
(filehandle plus deviceid?), so if we set an upper limit of 4k per
layout, that'd allow about 20 stripes (20 servers) per layout.  Is that
reasonable?

--b.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 7 Sep 2005 10:09:09 -0700

No, That is not seasonable at all. It must be in the hundreds.
Marc.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 7 Sep 2005 13:16:28 -0400

On Wed, Sep 07, 2005 at 10:09:09AM -0700, Marc Eshel wrote:
> No, That is not seasonable at all.

Why not?  Keep in mind this is only for a single layout.  You can still
have hundreds of data servers, and use different data servers for
different files.  I think you can even hand different layouts to
different clients for load balancing.

So you only need hundreds if you need to be able to give a single client
access to a single file through hundreds of data servers.

> It must be in the hundreds.

So if we made it 32k, that'd allow about 160 servers.  Would that do?

--b.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 7 Sep 2005 11:04:51 -0700

> different clients for load balancing.
 
Yes, I know, but I don't like the protocol to limit me if I want to use a 
larger set.

> So if we made it 32k, that'd allow about 160 servers.  Would that do?
No 160 will not be enough. We have today customers with clusters of few 
thousand nodes where at least few hundred nodes have access to the disks. 
We typically stripe our file as wide as possible so we can make use of all 
disks and all data servers for any file that will be big enough to span 
them. I can make use of hounders with todays systems so for a protocol 
that targets future system I would expect even more.
Since the FH is identical (for some implementations) it might be nice to 
have it only once in the layout. It can still be a FH for use only with 
data servers.
Marc.

---

From: Brent Welch <[email-masked]>
Date: Wed, 07 Sep 2005 11:18:48 -0700

We need to figure out what we can agree on here.

I would like to get the semantics of GETDEVICEINFO simple,
and I would like to see layout definitions that range from
very simple to possibly very complex.  The goal is that
everyone can supply a simple implementation, and once we
gain experience we can load in more complexity by defining
new layouts with richer semantics.

In my view (and my system) GETDEVICEINFO is managed by a
communication transport layer that
doesn't necessarily know much about the operations being performed
over that communication channel.  Essentially the RPC layer.
Do you guys have an RPC layer that knows how to manage multiple
addresses for the end points?  In practice, I do my multipath
routing at the IP layer with multiple routes to a destination network.
All the compensation for failed routes is handled by the IP layer.
I'm not a FiberChannel expert, but I believe they have a similar
strategy.  At the beginning of time (resource discovery) they figure
out what devices are out there, and use a kind of fingerprint to
reliably determine what they are talking with.  It is this fingerprint
information that is returned in GETDEVICEINFO.  My point is that
there is already a bunch of machinery down at this level, and I
think it will cause confusion if we try to add our own notion of
"multi-path" at this level that is somehow different than the
multi-path already supported by the network transport layers.

I think the following statement is perhaps at the core of the issue:

> would have to deal with yet another layout. 

I think the urge to have the grand unified layout that absorbs all features
is the wrong direction to go.  If you are trying to save work by doing
this, you'll actually create a very complex layout that only works,
in practice, in a few stylized ways.

Again, my goal is simplicity.  We need to agree on the core layouts
that are simple enough that everyone can implement them reliably. 
If you really, really want the notion of redundant devices, then
by all means add those semantics to your layout.  But please, can you
leave it down inside your layout and not make GETDEVICEINFO more
complicated?  Is that really too much to ask?

www.panasas.com
[email-masked]

---

From: Brent Welch <[email-masked]>
Date: Wed, 07 Sep 2005 11:25:14 -0700

Keep in mind that some applications have single files that are
multi-terabytes in size, and so distributing them over many, many
servers may be just what you want to do.

I think Garth has put a parameter in that specifies how much
memory the client has for the returned layout.  I think that
is sufficient - I'd hate to see a fixed upper bound on the size
imposed by the spec.

>>>"J. Bruce Fields" said:

www.panasas.com
[email-masked]

---

From: Todd Pisek <[email-masked]>
Date: Wed, 07 Sep 2005 14:43:06 -0500

I'm a newby here in pNFS land, but I have some experience
in networking as well as SANs. There are a multitude of
reasons, IMO, that GETDEVICEINFO should only concern
itself with endpoints and not get involved with path
management.

There is no assurance that pNFS meta data servers
have the same visibility to the data servers as the
clients. This means the meta data server would need to
obtain path information for every requesting client. This
information will already be available to the client as
part of the normal SAN operation.

In the realm of FC SAN, multi-port storage controllers
have vendor specific semantics associated with port
access. The meta data servers would need to maintain
this information on a per data server port basis. Again,
this is already provided by SAN path management.

There is a high probability that the network connecting
clients to data servers will implement some form of QoS,
traffic shaping, etc. within the network. In a multi-path
environment, path selection will play a part in meeting
the SLA's. IMO, the meta data server only has a part to
play WRT endpoint selection and scheduling, when considering
QoS issues.

As a general architectural point, I believe the main motivation
for pNFS is to separate control from data. Allowing the meta data
server to partake in path selection begins to muddle data and
control.

--- Todd

> We need to figure out what we can agree on here.
:

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 7 Sep 2005 16:14:17 -0400

On Wed, Sep 07, 2005 at 11:25:14AM -0700, Brent Welch wrote:
> memory the client has for the returned layout.

I see that LAYOUTGET has a "maxcount" parameter, and the server can
return TOOSMALL errors.  So a client can retry with increasingly large
buffers.  Does a client that wants to interoperate with any pNFS server
need to be prepared to retry with an arbitrarily large buffers?

Are there reasons why the NFS protocols have traditionally allowed
clients to set firm upper bounds on the sizes of rpc's?  I think the ACL
attribute is the one case where NFSv4 breaks with that rule, and I find
it somewhat awkward.

> I think that is sufficient - I'd hate to see a fixed upper bound on
> the size imposed by the spec.

OK.  But I'm slightly confused then by your original claim that "there
generally isn't a problem with big layouts in the files or objects
case."  Either the client has to be ready to handle big layouts or not.

--b.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 7 Sep 2005 13:50:12 -0700

> complicated?  Is that really too much to ask?

I already said that option 2b, which will put the alternate devices in the 
layout, will be fine too. Maybe Dave or Garth can say why they preferred 
option 2a. Now what do you mean "your layout"? If you mean a second file 
layout I would disagree. This can be an optional extension only to the 
existing file protocol layout. Ignoring it, which any server or client 
can, will keep your implementation as simple as if the option did not 
exist. So I think that the core issue is why you think that an option that 
can be ignored will make implementation more complex.
Marc.

---

From: Brent Welch <[email-masked]>
Date: Thu, 08 Sep 2005 14:05:14 -0700

> need to be prepared to retry with an arbitrarily large buffers?

Not necessarily.  Even if my multi-terabyte file is distributed over
1000's of data servers, I could get back a much smaller map that just
provides a multi-gigabyte window into that file.   So, the model is not
to retry until you get a multi-terabyte layout, but to do your I/O
in smaller ranges of the file.  You may need to be creative in your
layout definition to do that efficiently, but in the worst case of a
64K stripe unit spread over 10 million data servers, the server
could give out a layout for 64 Meg that listed 1000 servers.
Now, I wouldn't implement a layout like that, partly for this reason,
but the client could make forward progress.  I would only expect TOOSMALL if
the client-supplied buffer were just a handful of bytes or something.

 > case."  Either the client has to be ready to handle big layouts or not.

It's just that there can be an order of magnatude difference in the
size of a block layout map and a object or files layout map.

www.panasas.com
[email-masked]

---

From: Marc Eshel <[email-masked]>
Date: Thu, 8 Sep 2005 14:29:50 -0700

> the client-supplied buffer were just a handful of bytes or something.
The only problem with this approach is that if you can not fit all the 
data servers in to one layout that describes striped file you have switch 
to a one to one mapping of the file which can take many messages to 
describe for a very big file.
Marc.

---

From: Brent Welch <[email-masked]>
Date: Thu, 08 Sep 2005 16:01:47 -0700

> layout, will be fine too. Maybe Dave or Garth can say why they preferred

Super!  so we can keep GETDEVICEINFO simple.

 > can be ignored will make implementation more complex.

By "your layout", I meant "files layout" as opposed to "object" or 
"block" layout.  So, if the redundancy between devices is not part
of GETDEVICEINFO but rather a semantic of the files-based layout,
then that is fine with me.  However, I realize we need to explicitly
discuss the following issue:

Is the aggregation scheme is defined generically or essentially private
to the layout?  I suggest you look at draft-zelenka-pnfs-obj-01.txt
for a proposal how to define the aggregation scheme generically
so it could apply to blocks, objects, and files.  The document defines
these aggregation types:

   enum pnfs_aggregation_type4 {
     PNFS_AGGREGATION_TYPE_SIMPLE                  = 1,
     PNFS_AGGREGATION_TYPE_STRIPED_SPARSE          = 2,
     PNFS_AGGREGATION_TYPE_STRIPED_DENSE           = 3,
     PNFS_AGGREGATION_TYPE_MIRRORED                = 4,
     PNFS_AGGREGATION_TYPE_STRIPED_MIRRORED_SPARSE = 5
     PNFS_AGGREGATION_TYPE_STRIPED_MIRRORED_DENSE  = 6
   };

   union pnfs_layout_aggregation_map4 switch (pnfs_aggregation_type4) {
     case PNFS_AGGREGATION_TYPE_SIMPLE:
       pnfs_layout_componentid4        component;

     case PNFS_AGGREGATION_TYPE_STRIPED_SPARSE:
     case PNFS_AGGREGATION_TYPE_STRIPED_DENSE:
       length4                         stripe_unit;
       pnfs_layout_componentid4        components<>;

     case PNFS_AGGREGATION_TYPE_MIRRORED:
       pnfs_layout_componentid4        components<>;

     case PNFS_AGGREGATION_TYPE_STRIPED_MIRRORED:
       length4                         stripe_unit;
       uint16                          mirror_cnt;
       pnfs_layout_componentid4        components<>;
   }

By aggregation scheme, we
mean the math used to go from a file offset to which device
and where in that device to access.  I would also include 
semantics like your redundant filer head devices as a new kind
of layout/aggregation scheme where you can read or write from any device.
So, by "your layout", I'm assuming a new case in the
pnfs_layout_aggregation_map4 union that lists an array of
components.  It's a lot like MIRRORED, except is has the property
that the client writes to one component and it shows up on all
of them.

Finally, we get to the issue of how many of these different
layouts and aggregation schemes do clients and servers have to
support?  Are there "mandatory" pNFS layout types?  The
server, of course, needs only implement the ones it wants to.
The client can also always fall back to READ/WRITE to the
file server, so it has a fallback position, too.

What I'd like to avoid is that there is a single layout/aggregation
scheme that has a bunch of internal options about how it actually
works.  It is probably a fine point, but it seems much cleaner
to support or reject a layout type as a whole, rather than 
dig into the parameters associated with the layout and decide.

www.panasas.com
[email-masked]

---

From: Brent Welch <[email-masked]>
Date: Thu, 08 Sep 2005 16:18:52 -0700

> describe for a very big file.

Right, we don't use that for our really widely striped files.
Instead, we use a two-level scheme where you stripe the
first N gigabytes over M servers with a traditional striping
pattern, and then shift to another M servers for the next
N gigabytes, and so forth.  An advantage of this is that it
lets clients focus their attention on a smaller number of
data servers.  One client can't effectively draw data from 1000
servers at once, at least in our experience. And, we actually
do give out the complete map, even if it covers 1000 servers.
If you wanted to give out fewer than the complete set of servers
in the layout, then you'll supply the initial offset so
the client can do the math right.  Going back to the other
thread on "equivalent servers" and different layout and aggregation
schemes, this is yet another aggregation scheme for dealing 
with really large files.

www.panasas.com
[email-masked]

---

From: Garth Goodson <[email-masked]>
Date: Thu, 08 Sep 2005 16:32:38 -0700

>        pnfs_layout_componentid4        components<>;

Initially there was a suggestion to make the aggregation scheme(s) 
generic and push them to the generic layout.  There has been some 
pushback to that (as I don't think we can all agree...).  As well, at 
the moment, I would like to only commit to a generic striped layout 
(talking about adding a mirrored layout with out-of-band communication 
makes me uneasy at this point in time).  Thus, I think it is more 
sensible, at this time, to place the aggregation type specific 
information in each layout.

The file-based layout (and I can imagine the same with the block based 
layouts) will (at first) have only one aggregation scheme (striped) and 
will not have any reference to any other type of scheme (e.g., 
mirrored).  The structures will probably reflect those that currently 
exist in the -03 version of the spec.  Unless we wanted to add another 
level of indirection by allowing other types but only defining striped.

-Garth

---

From: Mike Eisler <[email-masked]>
Date: Thu, 8 Sep 2005 19:57:01 -0700 (PDT)

> should remove this reference, or clarify it).  We have avoided defining 

Please remove it.

> framework.  It could be added later as an additional protocol (such as 

Let's not do that.

>From the perspective of secure environments,
blocks has a fairly clear (and limited) deployment scenario, whether pNFS
is being used or not. Objects are somewhat less limited in the number of scenarios
where they can be deployed. NFSv3 data servers, even with GSS authentication,
would be secure in fewer deployment scenarios
than non-parallel NFSv3 that has GSS authentication. But 
customers would assume: "hey, it is GSS-NFS, I can use it like I used GSS-NFS before".
And then they'd get compromised, and pNFS gets
a bad reputation.

---

From: Brent Welch <[email-masked]>
Date: Fri, 09 Sep 2005 09:13:54 -0700

>>>Garth Goodson said:

 > information in each layout.

OK with me.

 > level of indirection by allowing other types but only defining striped.

I don't think the level of indirection adds much.

www.panasas.com
[email-masked]

---

From: Marc Eshel <[email-masked]>
Date: Fri, 9 Sep 2005 10:04:54 -0700

> information in each layout.

This doesn't make sense to me. We should try to have as much as in common 
as possible. There are few different ways to describe the data layout and 
it can be shared by all protocols. What was the reason for the push back? 
you can define few different layouts that will make everyone happy and 
than agree on the required ones for each protocol. 
I don't know what you refer to by out-of-band communication. The alternate 
deviceid will not be any different than the primary deviceid. In the 
device list the server can return the list (id=1 DS=x, id=2 DS=y) and than 
the layout will return id 1, and 2 as an alternate. If you have any 
problems using 1 than use 2 you might already connected to 2 because it 
was used as the primary on a different layout.
 
> level of indirection by allowing other types but only defining striped.

I think that we need more than that. I would like to add a one to one 
mapping and something like the mirrored for the alternate deviceid. 

Marc.

---

From: Garth Goodson <[email-masked]>
Date: Fri, 09 Sep 2005 10:31:54 -0700

> than agree on the required ones for each protocol. 

I'm not sure I follow...

Currently we have three proposed layouts: files, objects, and blocks. 
There is one layout for each.  Files will initially use striping. 
Objects have striping + mirroring, but as I understand, require some 
out-of-band protocol for mirroring.  Blocks was just a one-one mapping, 
but thought striping would be useful.

The suggestion was to make striping generic.  However, some pieces of 
striping aren't wanted by other protocols (e.g., the dense/sparse stripe 
types for objects, and they don't make sense for blocks).

> was used as the primary on a different layout.
This is independent of the equivalent device ID discussion (if that's 
what you are getting at).

> mapping and something like the mirrored for the alternate deviceid. 
Initially, we are not going to have more complicated layout mappings 
than striping for files (it would delay the specification too much at 
this point).  A one-one mapping is simply a stripe with one data server.

If you would like to propose a mirrored layout, feel free to write up an 
ID and it will be discussed.  But as I said, in my opinion, I doubt it 
will make it into the first cut of the files layout.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Fri, 9 Sep 2005 11:11:37 -0700

> > than agree on the required ones for each protocol. 

I meant aggregation schemes.

> types for objects, and they don't make sense for blocks).
It looks like objects do have dense/sparse aggregation types.

> what you are getting at).

Why is it independent? What do you think should be done for alternate 
devices.

> will make it into the first cut of the files layout.

Not mirrored but something similar to be used for alternate devices. I am 
not sure how this process work, but yes I can write a more detailed 
proposal.
Marc.

---

From: Garth Goodson <[email-masked]>
Date: Fri, 09 Sep 2005 11:37:59 -0700

To make a fully generic layout aggregation scheme, we would have to do 
something like this (very roughly):

    /* Per component info */
    struct pnfs_component_file {
        nfs_fh4                 fh;
        pnfs_deviceid4          dev_id;
    };

    union pnfs_layout_component4 switch (pnfs_layout_type) {
      case PNFS_NFSV4_FILE_LAYOUT:
         pnfs_component_file      file_component;
      case PNFS_BLOCK_LAYOUT:
         pnfs_component_block     block_component;
      case PNFS_OBJECT_LAYOUT:
         pnfs_component_object    object_component;
    };

    /* Aggregation of components */
    union pnfs_layout_aggregation_map4 switch (pnfs_aggregation_type4) {
      case PNFS_AGGREGATION_TYPE_SIMPLE:
        pnfs_layout_component4        component;

      case PNFS_AGGREGATION_TYPE_STRIPED_SPARSE:
      case PNFS_AGGREGATION_TYPE_STRIPED_DENSE:
        length4                       stripe_unit;
        pnfs_layout_component4        components<>;

      case PNFS_AGGREGATION_TYPE_MIRRORED:
        pnfs_layout_component4        components<>;

      case PNFS_AGGREGATION_TYPE_STRIPED_MIRRORED:
        length4                       stripe_unit;
        uint16                        mirror_cnt;
        pnfs_layout_component4        components<>;
    };

    /* Generic structures for layout types */
    union pnfs_layout_types switch (pnfs_layout_type) {
      case PNFS_NFSV4_FILE_LAYOUT:
         pnfs_layout_file_generic      file_layout;
      case PNFS_BLOCK_LAYOUT:
         pnfs_layout_block_generic     block_layout;
      case PNFS_OBJECT_LAYOUT:
         pnfs_layout_object_generic    object_layout;
    };

    /* Main layout structure */
    struct pnfs_layout {
        length4                      length;
        offset4                      offset;
        iomode                       iomode;
        pnfs_layout_types            layout;
        pnfs_layout_aggregation_map  aggr_map;
    };

I have a couple of concerns:  By moving the aggregation type to the top 
level, do we need to define common protocol for each aggregation/layout 
combination (e.g., we need a mirroring protocol that works for both 
files, objects, and blocks)?  If so, then things start getting tricky. 
Everyone has to agree on each of these protocols, and they may 
fundamentally work differently for each layout type; if this is the 
case, then how does it help having them at a common level?  Also, a 
number of these aggregation types won't be defined for certain layouts. 
  E.g., at first mirroring will not be defined for files, and probably 
not for blocks, sparse striping may not be useful for objects, etc.

Overall, I think that by moving the aggregation scheme up, it must mean 
that the method for performing that aggregation is common across layout 
types.  I'm not convinced it will be...

-Garth

Marc Eshel wrote:
>>>Garth Goodson <[email-masked]> wrote on 09/08/2005 04:32:38 
> 
> PM:
> 
>>>
>>>This doesn't make sense to me. We should try to have as much as in 
> 
> common 
> 
>>>as possible. There are few different ways to describe the data layout 
> 
> and 
> 
>>>it can be shared by all protocols. What was the reason for the push 
> 
> back? 
> 
>>>I don't know what you refer to by out-of-band communication. The 
> 
> alternate 
> 
>>>device list the server can return the list (id=1 DS=x, id=2 DS=y) and 
> 
> than 
> 
>>>problems using 1 than use 2 you might already connected to 2 because 
> 
> it 
> 
>>>>layouts) will (at first) have only one aggregation scheme (striped) 
> 
> and 
> 
>>>>level of indirection by allowing other types but only defining 
> 
> striped.
> 
>>>

---

From: Marc Eshel <[email-masked]>
Date: Fri, 9 Sep 2005 16:23:03 -0700

>     struct pnfs_component_file {

I will be happy with the following change and few words that describe it 
as an array of devices where the first one should be used but if there is 
a problem using the first one use the second one. I don't think this calls 
for a new ID.
 
                pnfs_deviceid4          dev_id<>;

> not for blocks, sparse striping may not be useful for objects, etc.

The only changes to the common draft is that you define more of the 
structure that are common for all the protocols. So instead of having the 
complete layout as opaque for none file layout you define more of it and 
only where it is different like different object type you would define as 
opaque. More common statures would allow for more common code. 

> types.  I'm not convinced it will be...
You already allow for opaque information on all common structures which 
implies different method to handle. If you want to be strict than there 
should be nothing in common between the protocols. All that we are doing 
is pushing the opaque information down one level. 
Marc.

---

From: Brent Welch <[email-masked]>
Date: Tue, 13 Sep 2005 11:47:55 -0700

>     struct pnfs_component_file {

I like everything up to this point, although we'd want to
decide on a smaller set of aggregation types for the first round.
Marc would want AGGREGATION_TYPE_REDUNDANT, for example.
And, as pointed out elsewhere, we may not need to call out SIMPLE
but instead use striping with one component.

 >          pnfs_component_object    object_component;

The use of the term "LAYOUT" in the above structure is confusing.
I would use PNFS_FOOBAR_COMPONENT instead of PNFS_FOOBAR_LAYOUT
to allow that component type to be used in various aggregations.

 >         pnfs_layout_component4        components<>;

I'd add PNFS_AGGRETATION_TYPE_REDUNDANT: right here,
as it has the same information as mirrored, but the semantics
are different in that you'd write-any/read-any with Marc's
redundant layout instead of write-all/read-any as in a
traditional mirror.

 >          pnfs_layout_object_generic    object_layout;

I don't understand what the generic structure is being used for.

 >         pnfs_layout_aggregation_map  aggr_map;

If you drop the sub-layout from the above type, then I think it is close.

 > not for blocks, sparse striping may not be useful for objects, etc.

I think the above paragraph is of central concern, and I apologize if
the objections I've raised have blocked the notion of generic layout
handling.  I actually think generic layout handling is fine and good.
There are some contradictions, but I think they can be resolved.

First, one goal is to have the device-specific access method be
fairly well contained in a client module that can be easily "plugged in"
to some client-internal API.  This module should manage the connection
to a device and be able to do I/O to it.  It should understand the
results of GETDEVICEINFO in order to set up the connection.

Second, a goal should be to contain the buffer management and parallelism
associated with the "p" in pNFS into a module that understands the
aggregation scheme.  This layer would do the math and so forth, and
then call out to the device-specific module to do the component I/O.

We haven't really discussed the structure on the client, but we 
should do that so the protocol definition meshes well with what
we want to do in the client.

 > types.  I'm not convinced it will be...

I think that for some simple things it can be, and we should try.
The problem is that some complex aggregation schemes have complex
error handling or other non-standard semantics, and I think we are
all worried that will disrupt the base case we are trying to agree
on as a starting point.  Personally, I don't want to see a single
layout that optionally encodes lots of different semantics because
I think the client code will be messy.

www.panasas.com
[email-masked]