Subject: [nfsv4] pNFS and equivalent data servers

From: Garth Goodson <[email-masked]>
Date: Wed, 31 Aug 2005 11:44:53 -0700

There has been some private email floating around about the desire for 
pNFS to enable the client to use alternative data servers to serve the 
same data stripe.

This can be useful in the case of a data server failure.  In this case 
the metadata server need not recall all layouts identifying the failed 
data server.  Instead, clients can automatically switch to using a 
different data server.

I believe that there are three main options.  I believe consensus is 
concentrating around option 2.  Feedback is welcome.

1) Don't change the spec; leave it to the metadata server to recall 
layouts or give layouts with the appropriate IDs for load balancing, 
etc.  Failure cases would require the client to write through the 
metadata server, or possibly return the layout, request a new one and 
depend on the metadata server handing out layout with a set of different 
devices (e.g., depend on the metadata server to remap the layout).

2) Change the pNFS spec to: (a) include a mapping of either one device 
ID to multiple dev_addrs or (b) provide a list of device IDs in the 
layout.  Either option would include the restriction that the devices 
identified are equivalent (i.e., can handle the same FH, stateid, 
logically share the same storage, etc).

Dave Noveck and I prefer 2a, while others prefer 2b.  I prefer 2a 
because it simplifies the layouts, can be used by other layout types, 
and can be handled by a lower layer (e.g., at the device level below the 
layout level).

Dave Noveck's comments on (2):
Requiring that these devices be equivalent would mean, I think, that
the basic purpose of this essentially multi-pathing and that in the
event of an IO error it doesn't provide additional capability.  If
the two data servers have sufficient connection that writing to one
is equivalent to writing to the other if there are multiple
copies, reading from one (if it is the only one the MDS gave you)
should give you access to all of the copies that there are.  In the
case of data server failure though, I can see that having a large
number of clients with a large number of layouts trade them in
might be troublesome for the MDS.  So having people switch on their
own would be a help.

3) Look at it from a coarser granularity and try to use/extend the 
location_info attribute applied to a data server.  This would require 
being able to request a list of 'equivalent' data servers, probably from 
the metadata server, which is different than the use for which it was 
initially designed.

This seems to be the least attractive option partly due to the increased 
complexity.  As well, it does not seem to be the best fit.

-Garth

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Marc Eshel <[email-masked]>
Date: Wed, 31 Aug 2005 13:32:10 -0700

I prefer option 2a too. Changing getdevicelist and getdeviceinfo to 
optionally return multiple dev_addrs per dev_id is a good idea for error 
recovery.
Marc.

---

From: Brent Welch <[email-masked]>
Date: Wed, 31 Aug 2005 15:39:28 -0700

I'm worried we are conflating three different things:
multiple paths to the same device,
redundant or "equivalent" data servers,
error recovery.

In addition, we are trying to optimize performance of an error case,
which should be viewed with suspicion. The important thing
to get right in the failure case is that storage is consistent,
and that the server implementation isn't horribly complex,
and that the optimization for the error case doesn't slow
down the non-error case.

Finally, you are also making some assumptions about the back
end that don't apply everywhere.  For example, the analog of
a file handle in the object world is a capability, and they are
by definition specific to a device.  You simply can't give them
to the wrong device and have them work.  I can also imagine
there could be files-based pNFS back-ends where the servers don't
understand each others FH.

Therefore, I think:

If you have multiple paths to exactly the same device, then
it is appropriate to put that information into the address(s)
returned with GETDEVINFO, and I'd be fine with that in the spec.
I'll note that this can also be viewed as a routing issue, and
you can have multiple paths (i.e., multiple routers) to get
between two IP addresses without needing multiple IPs at the
end points.  (In otherwords, multipath routing can be a matter
confined entirely to the networking configuration.)

However, if you are layering pNFS over a cluster file system
(remember, one goal is to not go through a file system, but
go to the storage devices they use) and it so happens that
different "devices" understand the same file handles, then
that is a private matter for your layout type.  It just isn't
generic enough to impose on everybody.  If you do anything,
I think it must be 2(b), and it is a new files layout type where
there are multiple, distinct devices that are "equivalent"
in Dave Noveck's terms. 

Let's go back to the error handling case because it is important.
Our strategy is that clients notify the server when they encounter
errors.  Note that in a wacked out customer network configuration,
the server-to-server visibility can be different than the
client-to-server communication paths. (I.e., your heartbeat
scheme may not detect the same problems your clients see.)
In response, our servers make their own judgment about the health
of the devices.  They do extra work to verify that the file is
in good shape.  They may in fact choose to give the client a
different layout that reflects whatever error recovery has
occurred (or is in progress).  Or, they may conclude that storage
is fine and the client is just confused, and so not recall any
layouts. This may sound long winded or overly general.
But, my assertion is that you cannot be too
careful, and any "optimizations" you are tempted to do in the
error case will result in corrupt data at some point.  Or, it
means that the metadata server has to revalidate storage all
the time, which will make normal-case performance slow.  I've
explored both alternatives :-/

Hmm - I probably went off a bit on the error recovery.  A simpler
way to state this is:

1. GETDEVICEINFO is about associating an address (or addresses) with
a storage device.  Overloading the semantics of what it means
to have multiple addresses is not appropriate.  All it means is
that if you send a message to any of the addresses it gets to
the exact same device.

2. Layouts that support redundancy are cool and useful.  I'm all
for including them in the spec as different kinds of layouts.  However,
error handling strategies for these schemes are likely to be
very specific to the semantics of that particular layout, and
trying to apply those semantics to all layouts is unlikely to work.

>>>Marc Eshel said:

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]