Subject: Re: [nfsv4] pNFS MDS and DS sharing clientid

From: <[email-masked]>
Date: Wed, 4 Jan 2012 16:59:51 -0500

> It seems that a bunch of stuff was put in the EXCHANGEID reply,
> like server_scope, so that "same/not same server" could be decided,
> but only EXCHANGEID gets that information.

It seems like you are thinking that there is something wrong with that,
and that you might need such information at other times.

When this was done, the assumptions was that one had already some occasion/
motivation to begin communication with the given server(s) and that you needed
server-owner information to avoid getting confused, as you would when two 
different IP addresses behaved as part of the same server, if you didn't have
the information that they were the same server.

I don't think that anyone then raised the question of a need for such
information at any other time.   

> But if I don't do what I suggested below, "How do I know whether or
> not the address in nfsv4_1_file_layout_ds_addr4.nflda_multipath_ds_list<N>
> is the "same server" as the MDS?

You don't.  Why is that a problem?

> Without a new EXCHANGEID, all I can see the client can do is compare
> the netaddr4 host IP address with the one provided at mount time for
> the NFSv4.1 (MDS) server. 

A matching IP address is a sufficient condition for server equivalence, so if you 
have equality, you know.  It is not a necessary condition, so in the event of 
inequality you don't know if it is the same server without an EXCHANGEID.  Why is 
that a problem?  

> And do I compare all of the ones in
> multipath_list4<>, looking for a match in this way?

Only if you had a very strong preference for the same server.  I might have a preference
to not use the same server (and look for a non-match) to give the MDS a performance break, 
But overall, it seems best to use the first address that works well and let the MDS, who
is in a better position to do global load balancing give you his best suggestion.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rick Macklem
Sent: Friday, December 30, 2011 12:41 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS MDS and DS sharing clientid

David Noveck wrote:
> > Now, the client has a netaddr for a DS server it needs to talk to
> > and
> > it already has an established clientid/session for the MDS.
> 
> I believe that it is best to treat this exactly the same as any other
> situation
> in which we have a session for a clientid for which USE_DS was
> returned.
> Whether USE_MDS was also returned doesn't/shouldn't affect the issue.
> 
Ok, I see what you are saying. I was just worrying about the 1st time
I needed to talk to a DS, but I can see that the same would apply to
subsequent cases of needing a clientid/session for an address in
nfsv4_1_file_layout_ds_addr4.

> > --> This was where I couldn't decide if/when I could use the already
> >     established TCP connection with clientid/session.
> 
> 
> I say that I the original EXCHANGED_ID got back USE_DS, and that
> session
> is still active, then you are done, and should use that session
> 
But if I don't do what I suggested below, "How do I know whether or
not the address in nfsv4_1_file_layout_ds_addr4.nflda_multipath_ds_list<N>
is the "same server" as the MDS?

Without a new EXCHANGEID, all I can see the client can do is compare
the netaddr4 host IP address with the one provided at mount time for
the NFSv4.1 (MDS) server. And do I compare all of the ones in
multipath_list4<>, looking for a match in this way?

It seems that a bunch of stuff was put in the EXCHANGEID reply,
like server_scope, so that "same/not same server" could be decided,
but only EXCHANGEID gets that information.

Still confused, but thanks for the help, rick

> If you did a new EXCHANGEID and the server was not the same, then
> there
> is a problem with the server, or it just happened to reboot in which
> case
> you would find that out when using the existing session.
> 
> 
> -----Original Message-----
> From: Rick Macklem [mailto:[email-masked]]
> Sent: Friday, December 30, 2011 12:03 PM
> To: Noveck, David
> Cc: [email-masked]
> Subject: Re: [nfsv4] pNFS MDS and DS sharing clientid
> 
> David Noveck wrote:
> > > If I understood it correctly (quite possibly not), this means
> > > that the server can handle both MDS and DS RPCs using the
> > > same ClientID replied by the EXCHANGEID Op.
> >
> > I agree.
> >
> > > If the above is correct, for the File layout case, how does the
> > > client know that the server in struct nfsv4_1_file_layout_ds_addr4
> > > reply for a GETDEVICEINFO is the same as the above MDS+DS?
> >
> > Why does he care about that question specifically?
> >
> > > (Is it simply "same IP address"?)
> >
> > If it isn't, then it is "same IP address or different IP address
> > connected to the same server (as shown by server owner)", but I
> > don't
> > know which of those two is right.
> >
> Ok, my question comes down to "How does the client know it's the
> same server", see below. (For the case where the MDS doesn't set
> EXCHGID_FLAG_USE_PNFS_DS it is not the same server.)
> 
> > The client needs to take the layout and establish a connection with
> > the DS.
> > So the first thing he does is to see if he has a session with that
> > address that has the USE_PNFS_DS bit set, and if he does, he uses
> > it.
> > (and he might also allow one to a different IP that is for the same
> > Server). I don't think that the fact that that server also set
> > USE_PNFS_MDS changes anything here.
> >
> > > Alternately, if I always do a separate EXCHANGE_ID, CREATE_SESSION
> > > on
> > > a different TCP connection to a DS in "struct
> > > NFSv4.1_file_layout_ds_addr4"
> > > (even if the MDS with EXCHGID4_FLAG_USE_PNFS_DS as above), will
> > > the
> > > server allow that and reply with a separate ClientID and SessionID
> > > that can be used for the connection to the DS?
> >
> > I expect that the server will give you back the same clientid that
> > you
> > got
> > the first time and you can use it do DS stuff (and MDFS stuff)
> >
> Ok, here's where I was stuck:
> - client connects to MDS and does EXCHANGEID, etc (Getting a reply
> with
> both EXCHGID4_FLAG_USE_PNFS_MDS and EXCHGID4_FLAG_USE_PNFS_DS set.)
> - at some point, the client needs to do I/O, so it does a
> LAYOUTGET followed by a GETDEVICEINFO for the deviceid4
> 
> Now, the client has a netaddr for a DS server it needs to talk to and
> it already has an established clientid/session for the MDS.
> --> This was where I couldn't decide if/when I could use the already
> established TCP connection with clientid/session.
> 
> From your comment above, is the correct thing for the client to do...
> - create a TCP connection to the netaddr4 in the reply from
> GETDEVICEINFO
> - do an EXCHANGEID4 on that TCP connection with the same
> eia_clientowner it used for EXCHANGEID on the MDS above
> --> Then use the reply from this EXCHANGEID to determine "same server"
> as the MDS above.
> - If same
> - just close this TCP connection and use the established one
> for the MDS above
> else (not same)
> - do a CREATE_SESSION and use this established session for
> the DS I/O operations
> 
> Does this variant sound right? (When I first posted, I had assumed the
> "same server" decision needed to be made before doing a new TCP
> connection
> and EXCHANGEID4.)
> 
> Thanks for the help, rick
> 
> > -----Original Message-----
> > From: [email-masked] [mailto:[email-masked]] On
> > Behalf
> > Of Rick Macklem
> > Sent: Thursday, December 29, 2011 8:25 PM
> > To: nfsv4
> > Subject: [nfsv4] pNFS MDS and DS sharing clientid
> >
> > I've read Sec. 13.1 of rfc5661 a couple of times and am still
> > confused (always:-), so I thought someone could help clarify
> > the case where the server replies with both:
> > EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS
> >
> > If I understood it correctly (quite possibly not), this means
> > that the server can handle both MDS and DS RPCs using the
> > same ClientID replied by the EXCHANGEID Op.
> >
> > If the above is correct, for the File layout case, how does the
> > client know that the server in struct nfsv4_1_file_layout_ds_addr4
> > reply for a GETDEVICEINFO is the same as the above MDS+DS?
> > (Is it simply "same IP address"?)
> >
> > Alternately, if I always do a separate EXCHANGEID, CREATE_SESSION on
> > a different TCP connection to a DS in "struct
> > nfsv4_1_file_layout_ds_addr4"
> > (even if the MDS with EXCHGID4_FLAG_USE_PNFS_DS as above), will the
> > server allow that and reply with a separate ClientID and SessionID
> > that can be used for the connection to the DS?
> >
> > Thanks in advance for any help, rick
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/nfsv4
> >
> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Rick Macklem <[email-masked]>
Date: Wed, 4 Jan 2012 20:57:12 -0500 (EST)

> and that you might need such information at other times.
Well, only if the client is expected to "know that the DS server
is the same one as the MDS server".

> > not the address in
> > nfsv4_1_file_layout_ds_addr4.nflda_multipath_ds_list<N>
See your previous reply, below.

> Only if you had a very strong preference for the same server.

Dave, for some reason we seem to be going in circles on this. In a
previous reply, you said:
>I say that I the original EXCHANGED_ID got back USE_DS, and that session
>If you did a new EXCHANGEID and the server was not the same, then there
>is a problem with the server, or it just happened to reboot in which case

I understood the above reply as saying; "The client should use the same
TCP connection/ClientID/session that it already got from the MDS, if it
is also the DS". And this implies that the client "knows it is the same
server", while you seem to agree that client does not know if it is the
same server.

> I might
> have a preference
This makes sense. However, this doesn't affect the case I am concerned with.

It is the case where the client chooses to use the DS that happens to also
be the MDS it is already communicating with, but doesn't recognize it as
"same server" (you agree, above, that it cannot), so it does a separate:
- TCP connect, ExchangeID, CreateSession using the DS address in the DeviceInfo
  reply. (It appears it can answer the "same server" question using the
  reply to ExchangeID, but not before that.)

Sorry that I don't seem to be expressing this clearly. I suspect it may be
"client implementor view" vs "server implementor view"?

rick

> not the address in
> nfsv4_1_file_layout_ds_addr4.nflda_multipath_ds_list<N>

---

From: <[email-masked]>
Date: Thu, 5 Jan 2012 07:13:14 -0500

Yes, we are going in circles and it is getting boring.  Sorry, everybody.

Rick and I seem to be in violent agreement even though we have retained sufficient
angular momentum to keep the discussion spinning.  I think I wrote some stuff that was
not clear and I will try below to reduce (and I hope not to increase) our angular
momentum.

<dn:ok> I say that I the original EXCHANGED_ID got back USE_DS, and that session
<dn:ok> is still active, then you are done, and should use that session

I would have been better off to just stop or say:

<dn:ok?> The above holds whether the EXCHANGE_ID also got back USE_MDS or not.

Instead, I led the discussion astray by saying.

<dn:sic> If you did a new EXCHANGEID and the server was not the same, then there
<dn:sic> is a problem with the server, or it just happened to reboot in which case
<dn:sic> you would find that out when using the existing session.

While it might have been clearer if I had said:

<dn:ok?> note that if you somehow ignored or forgot the fact that you had this
<dn:ok?> information from the previous EXCHANGE_ID, things would still work out
<dn:ok?> OK, and this also applies in all cases in which USE_DS was returned, whether
<dn:ok?> USE_MDS was also returned or not.
<dn:ok?>
<dn:ok?> In all such cases, as long as the server was operating correctly you would get
<dn:ok?> either:
<dn:ok?>    (1) the same clientid, server_owner, and use bits that you got in the
<dn:ok?>        original EXCHANGE_ID.
<dn:ok?>    (2) A new clientid indicating the server has rebooted since the first
<dn:ok?>        EXCHANGED_ID, in which case normal state/client handling procedures
<dn:ok?>        should be used to clear state associated with the old server
<dn:ok?>        instantiation and you can then act on the information in the new
<dn:ok?>        EXCHANGE_ID.

<rm:ok> I understood the above reply as saying; "The client should use the same
<rm:ok> TCP connection/ClientID/session that it already got from the MDS, if it
<rm:ok> is also the DS".

I din't say that but I agree that I did imply it.

But note that, while I didn't make it clear, what I meant was more like "The client
should use the same TCP connection/ClientID/session that it already got from the DS,
whether it is also the MDS, or not".  That implies your statement above but it would have
been clearer about the implied-knowledge issue which set us on our circular path.

<rm:sic?> And this implies that the client "knows it is the same
<rm:sic?> server", while you seem to agree that client does not know if it is the
<rm:sic?> same server.

It doesn't imply this.  You can, in analyzing a situation, make reference to things the
participants don't know, as long as you don't ask them to do something different,
based on knowledge they don't have.  In this case, although I should have made that
clearer, I am asking the server to use the connection/ClientID/session it got previously
whether it got it from the same server as the MDS or not, so such knowledge is not required,
as the client would do the same thing in either case.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rick Macklem
Sent: Wednesday, January 04, 2012 8:57 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS MDS and DS sharing clientid

rick