Subject: [nfsv4] pNFS data server multipathing

From: Dean Hildebrand <[email-masked]>
Date: Fri, 14 Dec 2007 14:12:48 -0800

Hello,

I'm reading section 13.6 in draft 17 and I am confused on the use of 
trunking.  I cannot tell if it complements pNFS or in competition with 
pNFS (another mechanism for parallel I/O).  If it complements pNFS then 
I would like to see more motivation in the spec than, "increase the 
speed of data transfer" since that is part of the motivation for pNFS.  
If it is competing as a method for performing parallel I/O, then the 
spec should decide on which mechanism it wants to support.

pNFS provides a client with a list of devices that it can stripe data 
across, i.e., parallel I/O.  Trunking seems to do this as well, could 
someone give more details on its purpose?

Back to section 13.6 in draft 17, we see multipathing useful for fail 
over and would like a way to indicate that trunking should not be used 
on the equivalent multipath servers.  The fail over server may not 
provide the same performance as the preferred server due to lack of 
prefetching, etc.  Clients that trunk data between equivalent multipath 
servers may see reduced performance for this reason. 
I also don't understand the last paragraph of 13.6,
"...Also, if multiple copies of the same data exist, reading from one 
must provide access to all existing copies. As such, it is unlikely that 
multipathing will provide additional benefit in the case of an I/O error."

Is this stating that multipathing should not be used for read replication?

Dean

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 14 Dec 2007 20:57:24 -0500

This may be orthogonal to your concerns but I think some of the lack of
clarity here may be due to terminology.  I went through other parts of
the spec (multi-server namespace particularly) and tried to distinguish
server addresses from servers.  I think where the text you are talking
about talks about "equivalent servers", it really should be talking
about multiple server addresses which connect to the same server (which
in implementation may be a cluster physically).

> I cannot tell if it complements pNFS or in competition with pNFS 
> (another mechanism for parallel I/O).  

I don't understand the distinction you are trying to make.  Since we are
talking about trunking of data servers, this presupposes use of pNFS and
so the competition, if it exists, is not a substantial one.

> If it complements pNFS then I would like to see more motivation in the

> spec than, "increase the speed of data transfer" since that is part of

> the motivation for pNFS.

If you want more or clearer motivation, then we can work on that but I
don't think we can or should have only one mechanism to meet particular
goals as improving speed of data or improving reliability.  
  
> If it is competing as a method for performing parallel I/O, then 
> the spec should decide on which mechanism it wants to support.

I don't agree at all.  Although pNFS and data server trunking are
complementary, the spec allows trunking of ordinary servers and that
might be competition for pNFS.  My guess is that will always lag pNFS,
but the spec's job is to provide the facilities and if someone figures
out how to build a high-performance clustered server that gives the
performance equivalent to that pNFS though multiple network addresses,
then more power to him.  If that happens, pNFS data servers will be less
used for a while, until they catch up and surpass this new more powerful
metadata server.  The spec should not decide that people should not make
faster metadata servers or that striping is no good.  It defines a
protocol that provides facilities for both and which ones are used will
depend on details of the implementation and the application, and
implementer's cleverness.

> pNFS provides a client with a list of devices that it can stripe 
> data across, i.e., parallel I/O.  Trunking seems to do this as 
> well, could someone give more details on its purpose?

If I want to get twice the bandwidth, I can divide the stripes up in a
finer fashion.  Or I can not do that and try to double the bandwidth to
each stripe.  Which I do will depend on details of the networking
configuration.  If my clients and servers each have two 10 gig
connections, then it makes sense to try to get that higher bandwidth.
When they upgrade to one 40 gig connection it won't. 

> Back to section 13.6 in draft 17, we see multipathing useful for 
> fail over and would like a way to indicate that trunking should not 
> be used on the equivalent multipath servers.  The fail over server 
> may not provide the same performance as the preferred server due 
> to lack of prefetching, etc.  Clients that trunk data between 
> equivalent multipath servers may see reduced performance for this 
> reason.

I think here the language may be adding to the confusion.  When you have
multiple addresses for the same server, then prefetching should be
effective across both addresses.  I think it is reasonable where you
have multiple network addresses that terminate in different servers
(they may have the same data stripes), to not use those in parallel but
only for failover. 

> I also don't understand the last paragraph of 13.6, "...Also, 
> if multiple copies of the same data exist, reading from one must 
> provide access to all existing copies. As such, it is unlikely that 

> Is this stating that multipathing should not be used for read
replication? 

It is saying that mutlipathing to one server is not the same as read
replication.  If you are doing read replication, then where one server
cannot read his copy, another might well be able to.  Where you are
multipathing to the same server, then where you can't read the data
(because the disk/loop/raid-group it is on is bad), trying to read from
the same server via a different network path is not likely to help you.

-----Original Message-----
From: Dean Hildebrand [mailto:[email-masked]] 
Sent: Friday, December 14, 2007 5:13 PM
To: [email-masked]
Subject: [nfsv4] pNFS data server multipathing

Hello,

Back to section 13.6 in draft 17, we see multipathing useful for fail
over and would like a way to indicate that trunking should not be used
on the equivalent multipath servers.  The fail over server may not
provide the same performance as the preferred server due to lack of
prefetching, etc.  Clients that trunk data between equivalent multipath
servers may see reduced performance for this reason. 
I also don't understand the last paragraph of 13.6, "...Also, if
multiple copies of the same data exist, reading from one must provide
access to all existing copies. As such, it is unlikely that multipathing
will provide additional benefit in the case of an I/O error."

Is this stating that multipathing should not be used for read
replication?

Dean

---

From: Marc Eshel <[email-masked]>
Date: Sat, 15 Dec 2007 09:49:54 -0800

Multipathing can be used for performance improvement and/or failure 
recovery depending on the server implementation. I think that we need a 
hint in nfsv4_1_file_layouthint4 to tell the client as to which option 
would be useful with the particular server. 
Marc.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 15 Dec 2007 15:21:46 -0500

I agree that we need some way to convey this to the client.

I think a hint might be complicated.  Consider the cases in which you
have addresses A, B, C, and D and A and B as a pair for higher bandwidth
can fail over to the pair of C and D which can also be used together.

So what I am thinking about is to drop the current restriction that the
addresses must be "equivalent" servers, which essentially mean that they
must be different addresses for the same server, and say that any
addresses that are for the same server (as shown by the server owner
returned by EXCHANGE_ID) can be used together for bandwidth aggregation
but that when two addresses designate different servers you should only
use them at a time in a failover mode.  You can use multiple addresses
for a given stripe at one time but can only use one server at a time and
may fail over from server to server.

So in he example above, A and B would have the same server owner as C
would with D, but the two pairs would have different server owners.

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]] 
Sent: Saturday, December 15, 2007 12:50 PM
To: Noveck, Dave
Cc: [email-masked]; Dean Hildebrand
Subject: RE: [nfsv4] pNFS data server multipathing

"Noveck, Dave" <[email-masked]> wrote on 12/14/2007 05:57:24 PM:

> This may be orthogonal to your concerns but I think some of the lack 
> of clarity here may be due to terminology.  I went through other parts

> of the spec (multi-server namespace particularly) and tried to 
> distinguish server addresses from servers.  I think where the text you

> are talking about talks about "equivalent servers", it really should 
> be talking about multiple server addresses which connect to the same 
> server (which in implementation may be a cluster physically).
> 
> I don't understand the distinction you are trying to make.  Since we 
> are talking about trunking of data servers, this presupposes use of 
> pNFS and so the competition, if it exists, is not a substantial one.
> 

> don't think we can or should have only one mechanism to meet 
> particular goals as improving speed of data or improving reliability.
> 

> then more power to him.  If that happens, pNFS data servers will be 
> less used for a while, until they catch up and surpass this new more 
> powerful metadata server.  The spec should not decide that people 
> should not make faster metadata servers or that striping is no good.  
> It defines a protocol that provides facilities for both and which ones

> are used will depend on details of the implementation and the 
> application, and implementer's cleverness.
> 
> 
> > data across, i.e., parallel I/O.  Trunking seems to do this as well,

> > could someone give more details on its purpose?
> 

> finer fashion.  Or I can not do that and try to double the bandwidth 
> to each stripe.  Which I do will depend on details of the networking 
> > may not provide the same performance as the preferred server due to 
> > lack of prefetching, etc.  Clients that trunk data between 
> I think here the language may be adding to the confusion.  When you 
> have multiple addresses for the same server, then prefetching should 
> be effective across both addresses.  I think it is reasonable where 
> you have multiple network addresses that terminate in different 
> servers (they may have the same data stripes), to not use those in 
> parallel but only for failover.
> 
> > multiple copies of the same data exist, reading from one must 

> (because the disk/loop/raid-group it is on is bad), trying to read 
> from the same server via a different network path is not likely to
help you.

> pNFS (another mechanism for parallel I/O).  If it complements pNFS 
> then I would like to see more motivation in the spec than, "increase 
> the speed of data transfer" since that is part of the motivation for
pNFS.

> prefetching, etc.  Clients that trunk data between equivalent 
> multipath servers may see reduced performance for this reason.
> access to all existing copies. As such, it is unlikely that

---

From: Mike Eisler <[email-masked]>
Date: Sat, 15 Dec 2007 15:53:40 -0800 (PST)

> I think a hint might be complicated.  Consider the cases in which you

Besides which, the layout hint is a write-only attribute.

> would with D, but the two pairs would have different server owners.

We need to finish this protocol.

The consequences of having data servers with completely different
server owners aren't fully baked. What does it mean from a security
perspective for example?

It seems to me if you want to convey data server topology (failover
only versus multipath only), the place
to do that is in the device address. But it isn't clear to me
we need that. 

> > > If it complements pNFS then I would like to see more motivation
> in 
> > > the
> > 
> > > spec than, "increase the speed of data transfer" since that is
> part 
> > > of
> > 
> > > the motivation for pNFS.
> > 
> the
> well,
> to 
> > > multipathing will provide additional benefit in the case of an
> I/O
> > error."
> > 
> the 
> data 
> fail 
> provide

---

From: Marc Eshel <[email-masked]>
Date: Sat, 15 Dec 2007 16:38:29 -0800

> may fail over from server to server.

That will also work for us.
Marc.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 17 Dec 2007 08:47:21 -0500

> We need to finish this protocol.

That's a sentiment I can agree with wholeheartedly.

> server owners aren't fully baked. What does it mean from a security 

But we already have this issue and I don't see how what I'm proposing
would make it worse.  Suppose you have meta-data server M and data
servers A, B, C, D,  A given file can be striped among A, B, C, and D
despite the fact that M, A, B, C, D have completely different server
owners.  It is the responsibility of one implementing the control
protocol to make sure that A, B, C, D act appropriately with respect to
security, which essentially means they would not do what M would not do.

Now I add data server E.  It has different server owner. No problem.

I assign a new stripe of the file to E.  No new security problem.

I now copy the stripes assigned to A and put them on E as a backup
(let's say this is read-only data, if it were writable the control
protocol implementer would have to provide for write co-ordination but
that is an implementation issue; the spec requirements stay as they
are).  There doesn't seem to be any new security issues.

The security equivalence/co-ordination issues here are between M and A,
M and B, etc., rather than between A and B, A and E, etc. The data
servers wind up with essentially the same security because they all have
to be the same as M, and this doesn't change that.

I'm going to have to rework this section anyway to make the terminology
match the rest of the spec.  Rather than speaking of servers that are
members of sets of equivalent servers, we need to talk about different
server addresses that are addresses to access the same server.  This
matches the approach elsewhere in the spec.  If anyone knows of other
problems in this regard let me know.  (Multi-server namespace excluded;
That's already changed in the CVS repository).

I'll prepare versions of this section with and without the substantive
modification but will only check in the one without the substantive
modification until we get consensus that this isn't going to add
complexity and delay.  Also the diff between the two will be
instructive.

> only versus multipath only), the place to do that is in the device 
> address. But it isn't clear to me we need that.

If we don't then I suppose the model would be that the MDS would recall
the layout and distribute new ones on request. 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Saturday, December 15, 2007 6:54 PM
To: [email-masked]
Subject: RE: [nfsv4] pNFS data server multipathing

> have addresses A, B, C, and D and A and B as a pair for higher 
> bandwidth can fail over to the pair of C and D which can also be used 
> together.
> 
> So what I am thinking about is to drop the current restriction that 
> the addresses must be "equivalent" servers, which essentially mean 
> that they must be different addresses for the same server, and say 
> that any addresses that are for the same server (as shown by the 
> server owner returned by EXCHANGE_ID) can be used together for 
> bandwidth aggregation but that when two addresses designate different 
> servers you should only use them at a time in a failover mode.  You 
> can use multiple addresses for a given stripe at one time but can only

> use one server at a time and may fail over from server to server.
> 

The consequences of having data servers with completely different server
owners aren't fully baked. What does it mean from a security perspective
for example?

It seems to me if you want to convey data server topology (failover only
versus multipath only), the place to do that is in the device address.
But it isn't clear to me we need that. 

> recovery depending on the server implementation. I think that we need 
> a hint in nfsv4_1_file_layouthint4 to tell the client as to which 
> option would be useful with the particular server.
> Marc.
>

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 17 Dec 2007 10:01:27 -0800

> That's already changed in the CVS repository).
If we keep trunking in the spec, then I agree with Marc that this 
approach will work for us.

But, I'm confused why would anyone use multiple ip addresses to the same 
server for performance reasons.  Where did this idea come from? If the 
goal is to utilize multiple nic's, then shouldn't they use bonding?  If 
the goal is to utilize multiple tcp streams, then that is an 
implementation issue that doesn't require multiple ip addresses (and 
CITI has already played around with the idea)

It seems to me that the idea of trunking should be at that the lower 
network layer, not at the higher layer of nfs.
Dean

>>> This may be orthogonal to your concerns but I think some of the
>>>       
>> lack
>>     
>>> of clarity here may be due to terminology.  I went through other
>>>       
>> parts
>>
>>     
>>> distinguish server addresses from servers.  I think where the text
>>>       
>> you
>>
>>     
>>> are talking about talks about "equivalent servers", it really
>>>       
>> should
>>     
>>> be talking about multiple server addresses which connect to the
>>>       
>> same
>>     
>>> I don't understand the distinction you are trying to make.  Since
>>>       
>> we
>>     
>>> pNFS and so the competition, if it exists, is not a substantial
>>>       
>> one.
>>     
>>> If you want more or clearer motivation, then we can work on that
>>>       
>> but I
>>
>>     
>>> particular goals as improving speed of data or improving
>>>       
>> reliability.
>>     
>>> complementary, the spec allows trunking of ordinary servers and
>>>       
>> that
>>     
>>> might be competition for pNFS.  My guess is that will always lag
>>>       
>> pNFS,
>>
>>     
>>> but the spec's job is to provide the facilities and if someone
>>>       
>> figures
>>
>>     
>>> performance equivalent to that pNFS though multiple network
>>>       
>> addresses,
>>
>>     
>>> less used for a while, until they catch up and surpass this new
>>>       
>> more
>>     
>>> should not make faster metadata servers or that striping is no
>>>       
>> good.  
>>     
>>> It defines a protocol that provides facilities for both and which
>>>       
>> ones
>>
>>     
>>> If I want to get twice the bandwidth, I can divide the stripes up
>>>       
>> in a
>>
>>     
>>> finer fashion.  Or I can not do that and try to double the
>>>       
>> bandwidth
>>     
>>> to each stripe.  Which I do will depend on details of the
>>>       
>> networking
>>     
>>> connections, then it makes sense to try to get that higher
>>>       
>> bandwidth.
>>     
>>>> fail over and would like a way to indicate that trunking should
>>>>         
>> not
>>     
>>>> be used on the equivalent multipath servers.  The fail over
>>>>         
>> server
>>     
>>> have multiple addresses for the same server, then prefetching
>>>       
>> should
>>     
>>>> provide access to all existing copies. As such, it is unlikely
>>>>         
>> that
>>     
>>> It is saying that mutlipathing to one server is not the same as
>>>       
>> read
>>     
>>> replication.  If you are doing read replication, then where one
>>>       
>> server
>>
>>     
>>> I'm reading section 13.6 in draft 17 and I am confused on the use
>>>       
>> of
>>     
>>> trunking.  I cannot tell if it complements pNFS or in competition
>>>       
>> with
>>
>>     
>>> then I would like to see more motivation in the spec than,
>>>       
>> "increase
>>     
>>> the speed of data transfer" since that is part of the motivation
>>>       
>> for
>> pNFS.
>>     
>>> across, i.e., parallel I/O.  Trunking seems to do this as well,
>>>       
>> could
>>     
>>> over and would like a way to indicate that trunking should not be
>>>       
>> used
>>
>>     
>>> multipathing will provide additional benefit in the case of an I/O
>>>       
>> error."
>>

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Mon, 17 Dec 2007 13:19:45 -0500

> network layer, not at the higher layer of nfs.

session or clientid trunking is useful for the nfs client state
implementation - associating client state to multiple connections, or to
multiple mount points. this is why i think it belongs above the network
layer.

-->Andy

Dean

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 17 Dec 2007 10:41:14 -0800

William A. (Andy) Adamson wrote:
>
>
> On 12/17/07, *Dean Hildebrand* <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
> implementation - associating client state to multiple connections, or 
> to multiple mount points. this is why i think it belongs above the 
> network layer.
Yes, buy why do we need multiple connections or multiple mount points to 
the same server?
Dean

>     > From: Mike Eisler [mailto: [email-masked]
>     <mailto:[email-masked]>]
>     > To: [email-masked] <mailto:[email-masked]>
>     > --- "Noveck, Dave" <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >
>     >
>     >> From: Marc Eshel [mailto:[email-masked]
>     <mailto:[email-masked]>]
>     >> Cc: [email-masked] <mailto:[email-masked]>; Dean Hildebrand
>     >> "Noveck, Dave" <[email-masked]
>     <mailto:[email-masked]>> wrote on 12/14/2007 05:57:24
>     >> PM:
>     >>
>     >>
>     >>> From: Dean Hildebrand [mailto:[email-masked]
>     <mailto:[email-masked]>]
>     >>> To: [email-masked] <mailto:[email-masked]>
>     >>> [email-masked] <mailto:[email-masked]>
>     >>> [email-masked] <mailto:[email-masked]>
>     >> [email-masked] <mailto:[email-masked]>
>     > [email-masked] <mailto:[email-masked]>
>     > [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Benny Halevy <[email-masked]>
Date: Mon, 17 Dec 2007 20:50:37 +0200

> Yes, buy why do we need multiple connections or multiple mount points to 

Sometimes server clusters share state without being able to bond their
nics into a network trunk.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 12:56:10 -0800 (PST)

--- Dean Hildebrand <[email-masked]> wrote:

> server for performance reasons.  Where did this idea come from? If

The limitations of existing NFS clients have been frustrating 
customers and applications for years.

> CITI has already played around with the idea)

NFSv4.1 session trunking doesn't stipulate different source IP
addresses, nor does it rule them out. NFSv4.1 sessions gets completely
away from IP addresses as a object of note in the protocol, and this
is a good thing.

I don't think it is an implementation issue based on my understanding
of how pre-NFSv4.1 reply caches work.

If anyone can explain how to trunk multiple connections without losing
exactly once semantics, and still work through NAT barriers,
I'm somewhat interested in reading about it. But only somewhat, as
I see little point in re-writing the sessions portion of the spec. 

> network layer, not at the higher layer of nfs.

Well it's a bit late to be expressing that idea.

Seriously folks, is anyone other than me interested in WG last call?

Did anyone pay attention when the Area Advisor made pointed remarks
about this WG's inability to publish anything?
 
> >> address. But it isn't clear to me we need that.

BTW, to expand on why I disagree with specifying different server
major IDs for fail over, the problem is that when the major IDs are
different, the client cannot make assumptions about a stateid created
against one server owner major ID working against another. Whereas 
if the major owner IDs match, the stateids work between the full server
owner.

``When doing client ID trunking, locking state is shared across
sessions associated with the same client ID. This requires the server
to coordinate state across sessions.''

So I oppose any changes in this area. If the WG decides we don't
need to enter last call this month, and want to delay for a month or
so, then I'll drop my opposition and review well thought out proposals.

> > the layout and distribute new ones on request. 

An even simpler model is that that the client attempts to make
connections and requests to data servers in the multi-path list.
If the client succeeds, the data server is available for fail over.
If the client fails, the data server isn't available right now. When
a data server stops responding, the client can attempt to
re-connect to one of the other erstwhile non-responding data servers.

Simple, and just requires some changes to spec prose. I'd assumed 
that's what clients would do anyway.

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 17 Dec 2007 13:39:33 -0800

Finishing the spec is definitely the priority.  I just think we need to 
make clear, and it seems to me that Dave's proposal would do this, what 
features are there for failover and which features are there for 
performance.  This will enable implementors to code appropriately and 
avoid situations where some exported file systems work really well while 
others degrade due to different interpretations of how to use specific 
features.

Dean

Mike Eisler wrote:
>>>> It seems to me if you want to convey data server topology
>>>>         
>> (failover 
>>     
>>> If we don't then I suppose the model would be that the MDS would
>>>       
>> recall
>>

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 14:07:13 -0800 (PST)

> performance.  This will enable implementors to code appropriately and

Dave's proposal prevents data servers from using stateids issued by
metadata servers.

> >> But, I'm confused why would anyone use multiple ip addresses to
> the
> >> same 
> >> goal is to utilize multiple nic's, then shouldn't they use
> bonding? 
> >> If

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 14:30:40 -0800 (PST)

> owners.  It is the responsibility of one implementing the control

I thought they don't have completely server owners. The major ID is the
same ... that's how we know that the stateids can be shared.

> only
> > "Noveck, Dave" <[email-masked]> wrote on 12/14/2007
> 05:57:24
> > PM:
> > 
> if

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 17 Dec 2007 20:35:06 -0500

> I thought they don't have completely server owners. The major I
> D is the same ... that's how we know that the stateids can be 
> shared.

That was not my intention, but I can see how you might assume that. 

Mu understanding that the global stateid constraint was something
that derives from the MDS-DS relatioship and was limited to the
stateids for the files which layouts assigned to a DS.  I would
well conceive of a deploment in which multiple disinct MDS's used 
a pool of DS's and divided up the stateid space so thaa the global
stateid approach could be used for each MDS together with the DS
pool without conflict.

Do you see any issues with that?  I can update the spec text to
make this clearer. 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Monday, December 17, 2007 5:31 PM
To: [email-masked]
Subject: RE: [nfsv4] pNFS data server multipathing

> protocol to make sure that A, B, C, D act appropriately with respect 
> to security, which essentially means they would not do what M would 
> not do.
> 

> The security equivalence/co-ordination issues here are between M and 
> A, M and B, etc., rather than between A and B, A and E, etc. The data 
> servers wind up with essentially the same security because they all 
> have to be the same as M, and this doesn't change that.
> 
> I'm going to have to rework this section anyway to make the 
> terminology match the rest of the spec.  Rather than speaking of 
> servers that are members of sets of equivalent servers, we need to 
> talk about different server addresses that are addresses to access the

> same server.  This matches the approach elsewhere in the spec.  If 
> anyone knows of other problems in this regard let me know.  
> (Multi-server namespace excluded; That's already changed in the CVS 
> repository).
> 

> recall the layout and distribute new ones on request.
>  
>

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 18:20:54 -0800 (PST)

> stateid approach could be used for each MDS together with the DS

If such MDSes are cooperating in that manner, then they probably share
the same major ID. E.g. they are organized in a cluster or a
federation.

That said, I can see value in sharing a DS among MDSes.

However, I don't think dividing up the stateid in order to allow
a single DS to be shared among many MDSes scales very well. The
more scalable way would be to encode the MDS identifier in the
data server filehandle. I vaguely recall the spec says that now
as an implementation suggestion.

I'm still uncomfortable with requiring two DSes to
have different major IDs or even differed major and minor
IDs in order to allow failover.  That just seems completely wrong.
What if I have two DSes that have the same server owner? Why 
shouldn't I be able to use the DSes as failover pairs if my storage
architecture is designed that way? 

If have to make a chance, I think it would be better to encode 
this in the layout. Either have an outer array of failover rows, 
and an inner array of multipatch columns, or the converse, or 
both, with flags indicating the outer array means.
(It is possible that only one hierarchy makes sense, sort like when
raid-0 and raid-1 are combined only one combination makes sense, but
I always forget what that combination is.)

> > > So in he example above, A and B would have the same server owner
> as
> > C
> > > > are talking about trunking of data servers, this presupposes
> use
> > of
> > > 
> > > > then more power to him.  If that happens, pNFS data servers
> will
> > be
> > > 
> > > > provide the same performance as the preferred server due to
> lack
> > of
> > >

---

From: Marc Eshel <[email-masked]>
Date: Mon, 17 Dec 2007 19:16:07 -0800

> as an implementation suggestion.

We use multiple MDSes and unless I am missing something I don't see any 
reason to do any stateid coordination among them. The MDS id is in the FH. 
The DS uses the backend protocol to check with the MDS from FH if the 
stateid presented is valid. 
 
> I always forget what that combination is.)

I though you wanted to keep it simple so we can finish this spec. I don't 
see what is wrong with some flag that tell the client what to use the 
multipathing for ?

Thanks, Marc.

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 17 Dec 2007 21:35:18 -0600

> see what is wrong with some flag that tell the client what to use the

The words and intent I have tried to communicate is that unless  
something
is broken that there be no more functional changes.

Is something broken and if so, how badly and if it isn't broken, why
are we having this discussion?

Spencer

---

From: Marc Eshel <[email-masked]>
Date: Mon, 17 Dec 2007 19:42:59 -0800

> are we having this discussion?
I think that we all agree that multipathing is useless without some 
clarification on how it can be used. All we need is some indication to the 
client if to use it for recovery or performance.
Marc.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 19:57:30 -0800 (PST)

> > If have to make a chance, I think it would be better to encode 

chance -> change

> see what is wrong with some flag that tell the client what to use the

Where would the flag be?

I think putting it in the EXCHANGE_ID results is the wrong place
to it.

I've already written that I don't think any change
is needed. The MDS returns a list of DSes, and the client uses
the ones that respond for multiple paths. When a path stops responding,
the client tries another DS that previously responded. If there is
response, the we have failover. If there isn't, we don't.

What I'm saying is that if consensus is to engage in feature creep,
then do it right. Server owners aren't designed for data server
failover. They are designed for MDS failover, but even then only
if an MDS redirects via an attribute. The logical place to perform
perform redirection is in the layout, not EXCHANGE_ID. 

Indeed MDS redirection effectively gives one another way to do 
data server failover. I don't think we have any text that talks about
what happens if an MDS that handed out layouts and
fs_locations_info attributes stops responding. My interpretation
would be that if the new MDS does not recognize the old MDS's layout
stateids, the client needs to get new layouts.

   -mre

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 20:08:20 -0800 (PST)

> client if to use it for recovery or performance.

I don't understand why it is always "or". If the data servers are
serving up read-only images, then the multi-path is clearly 
available for both recovery and performance.

It isn't clear to me under what conditions it isn't always
for both recovery and performance. Perhaps someone can
give a counter example.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 17 Dec 2007 23:40:14 -0500

> I don't understand why it is always "or".

But if it sometimes "or", we have an issue to resolve.

In the writable case, it would typically be "or" and there may be
readonly cases, as Dean has mentioned, in which readahead consideration
would mean that a client using multiple paths simultaneously would get
poorer performance.

Onw ay of resolving this is to use multipath for performance and allow
simultaneous use and eal with the failover case by requiring the
existing layout to be returned a new one assigned.  That would require
some way for the client to return a layout indicating that it is
returned because of access/availability issues.

The other way is to drop simultaneous use for performance and make this
for failover only.

You could also split this by minor server id.  If the minor id is the
same simultaneous use is allowed and if not it is for failover only.  

If we want don't want to allow multiple major server id's and don't want
to change the layout, which is always more complicated than it seems.
then we have to drop something, and what we drop can be prety small
compared to he benfit of getting pNFS going which we won't do if we
don't get to closure on this.  

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Monday, December 17, 2007 11:08 PM
To: [email-masked]
Subject: Re: [nfsv4] pNFS data server multipathing

--- Marc Eshel <[email-masked]> wrote:

> clarification on how it can be used. All we need is some indication to

> the client if to use it for recovery or performance.

I don't understand why it is always "or". If the data servers are
serving up read-only images, then the multi-path is clearly available
for both recovery and performance.

It isn't clear to me under what conditions it isn't always for both
recovery and performance. Perhaps someone can give a counter example.

---

From: Marc Eshel <[email-masked]>
Date: Mon, 17 Dec 2007 20:46:35 -0800

> available for both recovery and performance.

It doesn't have to always be "or" but we need a way to tell the client not 
to use it for performance because some configuration will give worst 
performance. 

> for both recovery and performance. Perhaps someone can

Dean wrote in the first note on this thread that the way we do internal 
locking and prefetch we expect the data to be read from the specified 
DSes. We can use other DSes in the cluster for failover but it will 
degraded performance.

Marc.

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 17 Dec 2007 22:47:59 -0600

> client if to use it for recovery or performance.

Given that client's have read-only replication in the form of a list
of servers and that seems to work okay for failover or performance  
(depending
on the client implementation choice) the layout multipathing seems just
as useful.

It seems, to me, that a new files based layout type might useful for  
this
type of additional information such that the client is informed to the
intent or purpose of the multipathing information.  Given that NFSv4.1
allows for new layout types, one can defined with the additional
information if it is found that the current layout type is deficient in
useful and specific ways.

Spencer

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 20:56:57 -0800 (PST)

> DSes. We can use other DSes in the cluster for failover but it will 

If some DSes are worse choices than others for failover and/or
performance then you need the type of logic we have into
fs_locations_info: FSLI4BX_READRANK, FSLI4BX_WRITERANK,
FSLI4BX_READORDER ,FSLI4BX_WRITEORDER.

Perhaps using server owners suffices for your server's implementation
but it doesn't suffice for the general problem.

---

From: Marc Eshel <[email-masked]>
Date: Mon, 17 Dec 2007 21:09:32 -0800

> on the client implementation choice) the layout multipathing seems just
 
You are ignoring the problem we described.

> useful and specific ways.

Why have another layout type when a flag is all we need. I don't want to 
cause anymore delays to the spec and if we don't resolve it than we will 
just not use multipathing, but lets try to do some simple change that will 
make it more useful.
Marc.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 21:18:30 -0800 (PST)

> would mean that a client using multiple paths simultaneously would

This a stretch. There is likely no expectation by a client doing round
robin among data servers serving the
same set stripe units is going to produce DS-side read ahead.

> returned because of access/availability issues.

Another way is for the pNFS server to offline DSes that to be used
for failover.

> same simultaneous use is allowed and if not it is for failover only.

You've convinced me we need flexibility in assigning server owners
to DSes. Requiring the DSes to share server owners if in a multipath
list seems inflexible.  

> to change the layout, which is always more complicated than it seems.

Messing with server owners is complex too.

> don't get to closure on this.  

How about we drop failover in DSes and use multiple MDSes to 
express failover.

Still, without ranking/ordering, I don't see how this solves the
issue Marc and Dean are articulating. Nor do I see how playing
with server owners solves it. The issue being that even in absence
of fail over in the file layout, some data servers in the multipath
list might be worse than others.

---

From: Mike Eisler <[email-masked]>
Date: Mon, 17 Dec 2007 21:20:21 -0800 (PST)

> just not use multipathing, but lets try to do some simple change that

What flag? Where would it appear?

---

From: Marc Eshel <[email-masked]>
Date: Mon, 17 Dec 2007 21:32:38 -0800

> What flag? Where would it appear? 

I already suggested to add a flag to nfl_util4.

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 18 Dec 2007 08:08:32 -0500

> FSLI4BX_READRANK, FSLI4BX_WRITERANK,

That's ordering once you have established failover.  Whether to use the
alternate locations simulatneously (i.e. for performance is indicated by
the simultaneous-use class).

It seems to me that if we are not to make this a whole lot more
complicated than we would be comfortable with (in terms of getting it
right the first time), I think the best direction to go is to use
multipathing for simultaneous use (i.e. performance).  For failover,
rather than tryng to encode the rules that client must follow as we have
to do for fs_locations_info since the server might be down, we can take
advantage fo the fact that the MDS is up (and if it isn't it has it's
own failover).  The client returns the layout indicating his access
problem, and then requests a new layout.  All the complicated logic
about what should fail over to what under what circumstances on what day
of the week is in the MDS and implementations can do what they want
without worry about the clients.  Let's not have the client make
decisions when we would be better off having the MDS making.  Failover
does not happen so often that this is a performance issue, so we can
take the hit of an MDS interaction for this case.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 06:06:10 -0800 (PST)

> I think the best direction to go is to use
> problem, and then requests a new layout. 

Works for me.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 06:19:25 -0800 (PST)

--- Marc Eshel <[email-masked]> wrote:

> > > Why have another layout type when a flag is all we need. I don't
> want
> > > to 
> > > cause anymore delays to the spec and if we don't resolve it than
> we
> > > will 
> > > just not use multipathing, but lets try to do some simple change
> that
> > > will 
> > > make it more useful.
> > 

I cannot find such an email. I did find an email where you
suggested a hint in nfsv4_1_file_layouthint4.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 18 Dec 2007 07:55:25 -0800

> suggested a hint in nfsv4_1_file_layouthint4. 
 
Right, nfl_util4 is a field in the nfsv4_1_file_layouthint4 struct, but I 
see that it is also included in thenfsv4_1_file_layout4 struct.
Marc.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 18 Dec 2007 09:05:38 -0800

> take the hit of an MDS interaction for this case.

If this is the consensus that this fine with me. It is ironic that I 
requested multipathing long time ago specifically for failover recovery 
but it got overloaded in a way that make it useless to us. If someone is 
planing to use it as is than fine nut otherwise we might as well remove it 
and simplify protocol, maybe save a tree or two.
Marc.

---

From: Dean Hildebrand <[email-masked]>
Date: Tue, 18 Dec 2007 11:51:18 -0800

I'm sorry it appears that we are realizing this problem late in the 
game, but I'm in the middle of implementing getdevicelist and 
getdeviceinfo on the server and found the problem.  The spec has changed 
so much from our original implementation that this is inevitable.  We 
want to finish the spec more than anyone, but the last bakeathon brought 
pNFS implementations (barely) to draft-13.  Be aware that problems will 
be found as we move to draft-18 at the next bakeathon.  As long as we 
don't add new features, hopefully that will be the last major painful push.

We are very open on how to resolve the distinction of failover - 
performance, just tell us which one is best:
1) Drop multipathing.  Client problems with a data server require 
retrieving a new layout from the server.
2) A flag as Marc suggested.
3) A clarification in section 13.6 that says (needs rewording) 
"multipathing is only for failover and that a client must access the 
first netaddr in nflda_multipath_ds_list and can only access subsequent 
netaddrs if a problem occurs."  This does not prohibit trunking, but 
limits its use with pNFS, which I think is a fine compromise.  Besides, 
if you are using multiple data servers, scaling bandwidth to any one of 
those data servers is useless as the available client bandwidth will 
already be fully utilized. (there are exceptions, but this is the most 
common scenario)
4) Another?

Dean

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 13:00:33 -0800 (PST)

> performance, just tell us which one is best:

Interested WG participants have to express opinions and it is
then up to the editors of WG chairs to interpret a rough 
consensus.

> already be fully utilized. (there are exceptions, but this is the

I don't follow your reasoning at all.

> 4) Another?

Dave Noveck also suggested that the failover use be dropped
from the layout, and the client be required to return its
layouts when it encounters a failure and get replacement
layouts.

Presumably this would only be done if all data servers serving
a stripe unit fail to respond. This is approach makes the
most sense to me, and is arguably merely implementation advice.

(It turns all returning/recalling layouts is probably not
necessary ... just change the device ID to device address
mappings.)

I think we have rough consensus that requiring data servers
in a multipath list to share the same major server ID is
a requirement we need to change.

Here is a suggested re-write of the multipathing section
<section title="Data Server Multipathing" anchor="file_multipath">
<t>
  The NFSv4.1 file layout supports multipathing to
  multiple data servers.
  Multipathing allows the client
  to switch to another data server that is exporting the
  same data stripe unit.
  Multipathing can be used for bandwidth scaling and/or
  higher availability in the event of server failure.
</t>
<t>
  To support data server multipathing, each element of
  the nflda_multipath_ds_list contains an array of one
  more data server network addresses.  This array (data
  type multipath_list4) represents a list of data servers
  (each identified by a network address).  The client is
  free to use any of the network addresses as a destination
  to send data server requests. If some network addresses are
  less optimal data paths to the data than others, then the
  MDS SHOULD NOT include those network addresses in an element
  of nflda_multipath_ds_list. If less optimal network addresses
  exist to provide fail over, the RECOMMENDED method to offer the
  addresses is to provided them in a replacement device ID
  to device address mapping, or a replacement device ID. When
  a client finds that no data server in an element of  
  nflda_multipath_ds_list responds, it SHOULD
  send a GETDEVICEINFO to attempt to replace the 
  existing device ID to device address mappings. If the MDS
  detects that all data servers represented by an element of
  nflda_multipath_ds_list are unavailable, the MDS SHOULD 
  send a CB_NOTIFY_DEVICEID (if the client has indicated it
  wants device ID notifications for changed device IDs) to
  change the device ID to device address mappings to the
  available data servers. If the device ID itself must be
  replaced, the MDS SHOULD recall all layouts with the
  device ID, and thus force the client to get new layouts and
  device ID mappings via LAYOUTGET and GETDEVICEINFO.
  
</t>
<t>
  Generally if two network addresses appear in
  an element of nflda_multipath_ds_list and both addresses
  have data servers listening, the two data servers
  are implementing client ID or session trunking (the latter
  is RECOMMENDED) as defined in <xref target="Trunking"/>,
  and this share the same server owner, or major ID of the server
owner.
  It is not necessary always for the two data servers to be trunking
  each other. For example the data could be read-only, and the
  data consist of exact replicas. 
</t>

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 18 Dec 2007 15:12:58 -0600

> most sense to me, and is arguably merely implementation advice.

Agreed.

> a requirement we need to change.

Agreed.

Text below looks appropriate.

Spencer

---

From: Robert Gordon <[email-masked]>
Date: Tue, 18 Dec 2007 15:20:54 -0600

> a requirement we need to change.

I agree.

[nit]

>  are implementing client ID or session trunking (the latter is
>  RECOMMENDED) as defined in <xref target="Trunking"/>,
>  owner. It is not necessary always..

"and this" ==> "and thus"

Robert.

---

From: Dean Hildebrand <[email-masked]>
Date: Tue, 18 Dec 2007 13:40:18 -0800

> I don't follow your reasoning at all.
Let me explain with an example.  You have 3 1Gbps data servers and 1 
1Gbps client.  The stripe width for the file is all 3 data servers and 
you have a stripe unit size of let's say 1MB (it doesn't matter). 

When reading or writing large files, each data server will only offer at 
most .33 Gbps, which will saturate the 1 Gbps client.  It should not be 
hard to realize .33 Gbps to each data server with a single connection.  
Using multipathing to a single data server will only split the .33 Gbps 
across the two connections, but will not affect the end-to-end bandwidth 
realized by the client. 

We have done extensive pNFS testing with 10Gbps clients and servers and 
the results are the same * 10.

Now the interesting scenarios come into play when mixing 10 Gbps clients 
with 1 Gbps data servers and with WAN connections. 

In the former, I have not seen any evidence in our experiments that 
multiple connections to from a 10 Gbps client to any one of several 1 
Gbps servers would help performance.  The first case is with <= 8 (or 
so) servers, in which case you will saturate performance of each data 
server with the 10 Gbps client.  In the second case where you have > 8 
or so data servers, then you are back to where we started with the 1 
Gbps clients and servers, where multiple connections do not increase the 
client's end-to-end banwidth.

With regards to a WAN, it is still a unknown question whether multiple 
tcp connections improve performance vs. a single tcp connection with a 
large buffer. Our experiments are mixed and all documentation I have 
found is about the same. One easy way to get multiple connections to a 
single server without multipathing is to just list each server ip 
address as a different data server in the nflda_multipath_ds_list array 
and change your stripe size accordingly.  But this is the less common 
path, whereas I believe everything is geared towards the most common way 
of doing things.

If you are imagining different scenarios, then please share.
Dean

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 14:02:54 -0800 (PST)

--- Dean Hildebrand <[email-masked]> wrote:
> >> Besides, 
> >> if you are using multiple data servers, scaling bandwidth to any
> one
> >> of 
> 1Gbps client.  The stripe width for the file is all 3 data servers
[...]
> If you are imagining different scenarios, then please share.

I'm imagining clients that have more network and network to
memory bandwidth than a single data server.

This is a big reason why pNFS is being invented: allow a single
client to stripe I/O across many data servers.

http://www.pdl.cmu.edu/pNFS/archive/gibson-pnfs-reqs.html

``Parallel NFS must enable high bandwidth access by single clients and
   aggregates of clients, especially clusters of clients, into one file
   system, into possibly small and arbitrary collections of files, and
   into just one file.''

Since we've apparently lost sight on the requirements, perhaps
we should make the requirements i-d a work item of this working
group.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 18 Dec 2007 14:03:24 -0800

>   MDS SHOULD NOT include those network addresses in an element

This just makes clear for us that we should not be using multipathing. I 
just wonder who out there finds it useful in this form and plans to use 
it, else we should just drop it.

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 18 Dec 2007 16:36:46 -0600

> we should make the requirements i-d a work item of this working

That is a good idea.  Given the difficulties we have had with
the RPC/RDMA drafts and over re-chartering, it would be helpful
to provide the context for the work and have the pNFS requirements
and design considerations as an informational RFC.

Any other opinions?

Spencer

---

From: Dean Hildebrand <[email-masked]>
Date: Tue, 18 Dec 2007 14:41:41 -0800

Mike Eisler wrote:
>>>> those data servers is useless as the available client bandwidth
>>>>         
>> will 
>>     
I explained these scenarios in my email.  The conclusion is that if 
multipathing is useless for a single client then it is even more useless 
when you have more clients than servers.  You can't get blood from a 
stone, i.e., more connections to an overloaded server does not help.  
Could you explain how multipathing improves performance in any of the 
scenarios I layed out or suggest a new scenario?

Dean

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 15:37:40 -0800 (PST)

--- Dean Hildebrand <[email-masked]> wrote:
 
> >>>> already be fully utilized. (there are exceptions, but this is
> the
> >>>> most 
> >>>> common scenario)
> >>>>     
> >>>>         

You and Marc have different ideas what "explain" means.

> stone, i.e., more connections to an overloaded server does not help. 

Presenting an example where a single client has 1/3 the total
network bandwidth capacity of a pNFS server doesn't win any
arguments with me. If that was only scenario the world had, then
we'd have run out of new oil reserves at least 5 years ago.

Hand waving an example where there is one 10 gb/sec client and
some unknown number of 1 gb/sec servers and no explanation of
physical configuration is even less persuasive.

Your scenarios assume storage devices with single network interfaces.
They tend to not exist in places where customers insist on redundancy,
but also insist on using that redundancy all the time. I think your
narrow view explains why Marc and you want to excise trunking from
the specification.

> scenarios I layed out or suggest a new scenario?

New scenario:

Client has 20 one X bit/sec interfaces.

pNFS server consists of five data servers, each with
two X bit/sec interfaces.

The data servers are named A, B, C, D, E.
Let us label the addresses of each interface of each
DS, .1 and .2, so A.1, A.2, B.1, B.2, etc.

The MDS presents device ID 0x123 which maps to:
 
struct nfsv4_1_file_layout_ds_addr4 {
        uint32_t        nflda_stripe_indices<>;
        multipath_list4 nflda_multipath_ds_list<>;
} daddr = { {0,1,2,3,4},  { {A.1. A.2}, { B.1, B.2}, ...
            { E.1, E.2 } {

If we drop multi-pathing the client gets to use just
5 X bits/sec . If we keep it, the client gets to use
10 X bits/sec.

---

From: Dean Hildebrand <[email-masked]>
Date: Tue, 18 Dec 2007 15:59:27 -0800

> 5 X bits/sec . If we keep it, the client gets to use
You talk about fail over, but you are failing over to the same physical 
server?  Not sure how that is useful.

If you are worried about performance with a high-bandwidth client, I 
covered this as well if you read the explanation a little closer.  Why 
not just give 10 data servers? You seem to have some attachment to 
having each item in  nflda_multipath_ds_list be a physical device. 

struct nfsv4_1_file_layout_ds_addr4 {
        uint32_t        nflda_stripe_indices<>;
        multipath_list4 nflda_multipath_ds_list<>;
} daddr = { {0,1,2,3,4,5,6,7},  { {A.1}, {A.2}, {B.1} {B.2}, ...
            {E.1}, {E.2} } 
(or any order you 

Both examples give the same performance but eliminating multipath 
simplifies the spec and the implementation. (and neither helps fail over)
Dean

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 16:23:33 -0800 (PST)

> server?  Not sure how that is useful.

First, failover is not necessarily the primary reason for the
topology I've presented; performance could be. Second, a single
network interface is single point of failure, so indeed failover
to the same physical server is useful. Third, it could be a separate
physical server that is, for example, sharing multi-ported disk
arrays with another physical server.

You should review the product offerings of existing NAS
vendors.

> covered this as well if you read the explanation a little closer. 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Not gonna play that game. You did not provide or address
my example before or now.

> Why 

[Why is your indices array only 8 units long?

I'll assume you meant 10. If you meant 8, then I give up.]

Because in my example the striping pattern is 5 units long and
in your example it is 10 units long. And in your example I have to
generate 10 filehandles for each file, and in my example, I have
to generate just 5. 

> Both examples give the same performance but eliminating multipath 

What if Santa Claus delivers me an extra network interface card for
each data server? You are going to make re-stripe to use the
my Christmas present? Are you a Grinch?

> simplifies the spec and the implementation. (and neither helps fail

Regarding not helping failover that's nonsense.

---

From: "Muntz, Daniel" <[email-masked]>
Date: Tue, 18 Dec 2007 16:28:01 -0800

He's not talking about failover; he's talking about performance
(redundant links, that are fully utilized).  Failed single link =
multipath link still up, but bandwidth reduced.  Customers like
redundancy, hate idle hardware.

I'd rather buy 5 NICs than 5 servers if I have bandwidth on the
backplane.

-----Original Message-----
From: Dean Hildebrand [mailto:[email-masked]] 
Sent: Tuesday, December 18, 2007 3:59 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS data server multipathing

> pNFS server consists of five data servers, each with two X bit/sec 
> interfaces.
>
> Let us label the addresses of each interface of each DS, .1 and .2, so

> A.1, A.2, B.1, B.2, etc.
>         multipath_list4 nflda_multipath_ds_list<>; } daddr = { 
> {0,1,2,3,4},  { {A.1. A.2}, { B.1, B.2}, ...
>             { E.1, E.2 } {
>
> 5 X bits/sec . If we keep it, the client gets to use 10 X bits/sec.
>   

struct nfsv4_1_file_layout_ds_addr4 {
        uint32_t        nflda_stripe_indices<>;
        multipath_list4 nflda_multipath_ds_list<>; } daddr = {
{0,1,2,3,4,5,6,7},  { {A.1}, {A.2}, {B.1} {B.2}, ...
            {E.1}, {E.2} }
(or any order you 

Both examples give the same performance but eliminating multipath
simplifies the spec and the implementation. (and neither helps fail
over) Dean

---

From: Marc Eshel <[email-masked]>
Date: Tue, 18 Dec 2007 16:34:00 -0800

> You and Marc have different ideas what "explain" means.

We are sharing with you our implemantation and performance testing and you 
prefer to stick to your ideas of how things should work and ignore our 
imput. I would like to hear from other implemators, but I suspect that no 
one cares about this feature and that is way they are quite. So again this 
feture is usless to us based on our implementation experience.
Marc.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 18 Dec 2007 19:55:35 -0800

> I'd rather buy 5 NICs than 5 servers if I have bandwidth on the

To achieve the above you use bonding of your NICs. This is a low level 
networking function that has no place is pNFS protocol.
Marc.

---

From: "Muntz, Daniel" <[email-masked]>
Date: Tue, 18 Dec 2007 21:04:49 -0800

Déjà vu.  I think we've looped on this point. 

> Sent: Tuesday, December 18, 2007 7:56 PM
> To: Muntz, Daniel
> Cc: [email-masked]; [email-masked]; Dean Hildebrand

---

From: Marc Eshel <[email-masked]>
Date: Tue, 18 Dec 2007 21:26:05 -0800

We are in this loop for sometime now, I still wait for someone to step 
forward and describe how they will be use multipathing, rather than how it 
might be possible to use it. I gave up on this feature few notes back, but 
I promise to try to break out of it now.
Marc.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 21:26:30 -0800 (PST)

> networking function that has no place is pNFS protocol.

Since trunking is part of the sessions layer, what you really mean
is that trunking has no place in the the NFSv4.1 protocol.

Trunking has been in the spec for at least a year and yours is the
first objection.

It appears to me like you'd be prefer to derail the process because
you've decided you don't want to support failover by re-mapping
device IDs and want it done *your* way.

http://en.wikipedia.org/wiki/Link_aggregation mentions some limitations
with link level aggregation:

`` Note that when using Microsoft Windows, establishing a trunk with
NICs usually only works among certain NIC types, and all must usually
be of the same brand.[citation needed] The trunk itself is typically
established at the device driver or NDIS level.
[...]

Typically the ports used in a trunk should be all of the same type,
such as all copper ports (CAT-5E/CAT-6), all multi-mode fiber ports
(SX), or all single-mode fiber ports (LX).[citation needed]

The ports also need to operate at the same speed. It is possible to
trunk 100-megabit ports together, but trunking a 100-megabit port and a
gigabit port together will most likely not work,[citation needed] even
though mixing port sizes within a trunk is technically supported in the
802.3ad standard. Ports operating in different duplex will not
aggregate. One half duplex and a full duplex port cannot aggregate.''

I also wonder how well link-level aggregation works through NAT. I
do know NFSv4.1 trunking works through NAT.

Sessions support simultaneous association of RDMA and non-RDMA
connections to a session, so link level aggregation would not
seem to substitute for that either.

Since NFSv4.1 doesn't pay much attention to target and source
network addresses, but does allow multiple connections, this means
that a client could potentially make multiple connections to a server,
each associated with the same client owner, major server ID pair,
and possibly the same session ID, without being aware of actually
network addresses involved. This is a strength of the new sessions
model.

Effectively, Marc and Dean would rather we prevent 
multiple connections from associating with a session, which would
make recovery from dropped connections and dropped requests take
longer. In order to recover, I have to wait for the server to figure
out the connection is dropped before it will let me associate a
replacement connection with the session.

I'm really surprised and disappointed that Marc and Dean see
no value in this aspect of sessions, but I'm not very inclined right
now to re-justify the value to them, or anyone else before WG last
call.

I think we should proceed with WG last call, and if Marc and Dean
and potentially others want to continue to demand the re-design 
of sessions and perhaps succeed derail everything, they can.

---

From: Marc Eshel <[email-masked]>
Date: Tue, 18 Dec 2007 21:50:26 -0800

> device IDs and want it done *your* way.

If you read more carefully my notes you might have seen that we stopped 
asking for any changes few notes back. If it was not clear I will say it 
again that we don't ask for more changes and we are not trying to derail 
any process. We are planing to do recovery the way that Dave suggested 
through the MDS. We were just continuing the discussion to share 
information like the fact that we are not going to use multipathing and 
wondering who does and how. I am done with this thread.
Marc.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 18 Dec 2007 21:57:55 -0800 (PST)

> asking for any changes few notes back. If it was not clear I will say

All I can see in your notes is "bonding".

> it 

Excellent.

> wondering who does and how. I am done with this thread.

Perfect.