Subject: [nfsv4] initial client connection(s) for pNFS

From: Rick Macklem <[email-masked]>
Date: Thu, 5 Jan 2012 11:26:40 -0500 (EST)

Ok, since Dave and I are going in circles, I'm going to try asking
the question again. (I think Dave was looking for some general case,
where I had simply intended to ask about the following scenario.)
(Dave, try and forget everything we've said until now, since this is
 a new question.;-)

The scenario:
A newly installed NFS client does the first NFS mount it has ever done.
- mount command to an NFS server specified by host IP address.
- client does TCP connect to server port 2049   - works
- client does NULL RPC to NFS version 4         - works, knows server supports v4
- client does Compound, minor vers=1, with EXCHANGE_ID op - works
  - client now knows that the server supports NFSv4.1
  - in the ExchangeID reply, there are the 3 flags
    EXCHGID4_FLAG_USE_NON_PNFS - clear
    EXCHGID4_FLAG_USE_PNFS_MDS - set - client knows that server can issue layouts, etc
    EXCHGID4_FLAG_USE_PNFS_DS - set - client knows that this server is also a DS
- client does CREATE_SESSION - works

- client does assorted NFS stuff, then
- client Opens a file, gets Open_StateID
  - since client knows the server is an MDS and the client supports pNFS, it then
  - client does LAYOUTGET - works (has never done this before) -
    - reply is for a files_layout and has deviceidX
  - client does GETDEVICEINFO with deviceidX - works
    - Now, the reply had a list of DS servers, by netaddr4
      (Lets for simplicity assume that each multipath_list4 is of length 1, so the
       client doesn't have a choice w.r.t. which one to use.)

Here's where I'm not sure what to do next...because I am not sure if a DS specified
in nflda_multipath_ds_list<> is the same server as the above MDS I am already connected
to.

So, what does the client do at this point?
#1 - just do a fresh TCP connection and EXCHANGE_ID for the address in nflda_multipath_ds_list<N>.
     and use that to talk to the DS.
OR
#2 - do the above, but check for same ClientID, scope and if the same as the above MDS,
     close this TCP connection and use the one for the MDS.
OR
#3 - compare the host IP address in nflda_multipath_ds_list<N> with the one used to
     establish the TCP connection to the NFS server above and if the same, just use
     the above established connection for DS operations. The problem here is "what do
     you do for the case where the host IP addresses are not the same"? Can you assume
     that means "use a new TCP connection/ClientID/Session"?

If this still doesn't make sense, don't worry about it. I'll try something and learn
what works. rick

---

From: Andy Adamson <[email-masked]>
Date: Thu, 5 Jan 2012 12:28:37 -0500

Hi Rick

> in nflda_multipath_ds_list<> is the same server as the above MDS I am already connected

The Linux client uses the same criteria for deciding if we already
have a connection set up (e.g. DS is same as MDS) as we do when we get
a second mount request to a non-pNFS NFS server.

If we have a connection where the NFS version, minorversion, protocol,
full socket address, and network namespace all match, we use it.

-->Andy

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Andy Adamson <[email-masked]>
Date: Thu, 5 Jan 2012 13:21:34 -0500

On Thu, Jan 5, 2012 at 1:12 PM, Nico Williams <[email-masked]> wrote:
> On Thu, Jan 5, 2012 at 11:28 AM, Andy Adamson <[email-masked]> wrote:
> This seems weak to me.  It'd be better if there was some higher-layer
> method for determining a server's identity.

All we currently have is what is returned in the file layout
GETDEVICEINFO operation - a netaddr4.

-->Andy

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Jan 2012 12:12:22 -0600

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 05 Jan 2012 14:05:09 -0500

> method for determining a server's identity.

There is: EXCHANGE_ID allows the client and server to mutually identify
each other by means of an exchange of unique identifier strings.
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 05 Jan 2012 14:09:43 -0500

> full socket address, and network namespace all match, we use it.

Yes, however we recognise that this is the wrong approach. Chuck is
implementing a better system (based on the EXCHANGE_ID identifier) as
part of the migration patches.

NetApp

---

From: Andy Adamson <[email-masked]>
Date: Thu, 5 Jan 2012 15:10:18 -0500

> part of the migration patches.

OK. Can you give an example of when the netaddr4 info returned by the
file layout GETDEVICEINFO can be the same for more than one server?

-->Andy

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 05 Jan 2012 15:32:35 -0500

> file layout GETDEVICEINFO can be the same for more than one server?

No, but the interesting case is when the netaddr4 info _differs_ from
the address we used to mount the MDS server with, yet EXCHANGE_ID
replies with the same owner and scope (i.e. network trunking). We don't
currently recognise that as being the same server.

FWIW, we have the same problem with the basic NFSv4.1 mount code: there
too we rely on the IP address instead of the EXCHANGE_ID results. As I
said, we do plan to fix this.

NetApp

---

From: <[email-masked]>
Date: Thu, 5 Jan 2012 19:35:44 -0500

> currently recognise that as being the same server.

Clearly you should recognize it as the same server.  If you don't, state management
can get confused, which is why this information was added in the first place.  Ignore 
it at your peril.

But even if you recognize that, this doesn't tell you what IP address you actually use.

For example, you could:
    1) use the netaddr in the info, unless it doesn't work well.
    2) prefer the mds address to the one in the device info when they are the same server.
    3) use network trunking on a common session involving the device info address and
       mds address if it is the same server.

There are even more options if any of the following are true:
     a) have multiple addresses in the device info 
     b) are already aware of multiple mds ip addresses that constitute the same server.
     c) are already aware of multiple ds ip addresses that constitute the same server.

> said, we do plan to fix this.

The issues are similar in this case.  Clearly, you want to use the EXCHANGE_ID
results for server identity.  

For what ip address you use, the range of choices is smaller in that, since the
MOUNT is giving you an Ip address there is no case for giving an existing MDS IP
priority over the mount ip address. I see two choices:
   1) use the mount ip address only.
   2) trunk the mount ip address and anything you know about that shows up as the
      same server.

As far as I know none of the options discussed here are prohibited by the spec, but
I haven't read it very carefully lately.  Other opinions on what is said by the spec
or what should be said by the spec in this area?

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Trond Myklebust
Sent: Thursday, January 05, 2012 3:33 PM
To: Andy Adamson
Cc: Rick Macklem; [email-masked]
Subject: Re: [nfsv4] initial client connection(s) for pNFS

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Thu, 5 Jan 2012 20:20:00 -0500 (EST)

Andy Adamson wrote:
[good stuff deleted for brevity]
> file layout GETDEVICEINFO can be the same for more than one server?
Sysadmin with fumbling fingers that accidentally assigns 2 servers the
same IP address, maybe? It'll get noticed, but maybe not before the servers
have been running for a while.

But, I agree with Trond that the "same server, different address" is the
far more likely case.

I'm going to re-read the section on EXCHANGE_ID now.

Thanks everyone for the feedback, rick

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Jan 2012 21:52:51 -0600

> But, I agree with Trond that the "same server, different address" is the

Also: NATs/proxies/firewalls.

---

From: Nico Williams <[email-masked]>
Date: Thu, 5 Jan 2012 21:53:51 -0600

Basically: don't use IP addresses in applications protocols if you can
at all help it.

---

From: Rick Macklem <[email-masked]>
Date: Fri, 6 Jan 2012 20:41:20 -0500 (EST)

> part of the migration patches.
I re-read the section of RFC5661 on EXCHANGE_ID4 (I should have done this
sooner, sorry for the noise;-) and, at least at this point, I think it is
the best way to handle the above.

Here's an interesting snippet of it:
   Note that the
   same client owner/server owner pair can have multiple roles.
   Multiple roles can be associated with the same client ID or with
   different client IDs.  Thus, if a client sends EXCHANGE_ID from the
   same client owner to the same server owner multiple times, but
   specifies different pNFS roles each time, the server might return
   different client IDs.
   (where multiple roles refer to MDS vs DS)

Since the above seems to say the same server can return a different ClientID
for the DS role than the MDS role, it seems that doing:
- make a new TCP connection to the server using the netaddr4 address
- do an EXCHANGE_ID4 on it, using the same co_owner etc, per "2." on
  pg. 509.
- then the client can check the reply for same ClientID from same server.
  - if same ClientID from same server
A)    - I think the client can choose to CREATE_SESSION for this new TCP
        connection to the ClientID and use it for DS operations
      OR
B)    - close the new TCP connection and use the one already established
        for DS operations

    At this point, it's not obvious which of these would be preferred.
    I can see an argument for:
    A) A separate session with as many slots as the server will support
       would permit more I/O operations to be done concurrently.
    B) because it uses less of the 16bit port# space.
    I am leaning towards A) at this point, but???

Anyone see problems with this, or have additional comments?

Thanks everyone, for your feedback, rick

---

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 12 Jan 2012 15:03:08 -0500

Rick,

In the windows pnfs client implementation we do pretty much the steps
you have outlined. However, we also check that netadd4 returned in the
deviceinfo list already matches something we've seen before (ie mds)
and then we skip sending the other exchange_id. However, we also do
the server matching after receiving a reply from an exchange_id in
case it's a multihome server.