Subject: Re: [nfsv4] NFS4 over VPN hangs when connecting > 2 clients

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 19 Mar 2012 12:28:51 -0400

On Mon, Mar 12, 2012 at 05:27:08PM -0400, J. Bruce Fields wrote:
> On Mon, Mar 12, 2012 at 05:14:16PM -0400, Chuck Lever wrote:
> > IMO, the server should do a comparison of the nfs_client_id4 strings,
> > and nothing else.
> 
> We're supposed to return CLID_INUSE when we see a setclientid from a
> "different" client using the same string, to keep clients from doing
> mischief with other clients' state (either maliciously or, as in this
> case, accidentally).
> 
> "Different" here is defined as "not having the same principal".  I know
> what that means in the krb5 case, but I'm less certain in the auth_sys
> case.

Cc'ing the ietf list.  Is it reasonable for a server to expect
setclientid's to come from the same client IP address at least in the
auth_sys case, or could that break multi-homed clients?

At least in the auth_sys case IP addresses are one of the only things we
have left to go on when the client's identifier-generation is messed up
(not that difficult).

--b.

---

From: Rick Macklem <[email-masked]>
Date: Mon, 19 Mar 2012 12:44:20 -0400 (EDT)

> auth_sys case, or could that break multi-homed clients?
I think that even a dhcp lease renewal might result in a different client
IP, if the client has been partitioned from the dhcp server for a while.

I'm not convinced that different client IP# implies different client.
(Even "same ip# implies same client" might not be true, if the dhcp
 server assigned the IP# to another machine while the client was partitioned
 from the dhcp server, I think? I haven't looked at current dhcp
 implementations, but it seems conceivable to me.)

For AUTH_SYS, all the FreeBSD server does is expect the same uid#.

rick
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Rick Macklem <[email-masked]>
Date: Mon, 19 Mar 2012 13:06:47 -0400 (EDT)

> implementations, but it seems conceivable to me.)
Oh, and what about the case of 2 clients that are sitting behind
the same NAT gateway? (I think they'd both be seen as having the
client host ip# of the gateway, but with different TCP connections
on different client port#s.)

> > At least in the auth_sys case IP addresses are one of the only
> > things
> > we

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 19 Mar 2012 13:36:56 -0400

> > IP, if the client has been partitioned from the dhcp server for a

Yeah, but by that point the client's v4 lease is probably expired anyway
so the client's not likely to be bothered by the NFS4ERR_INUSE.

> on different client port#s.)

Well, sure, but all I'm proposing here is returning NFS4ERR_INUSE in the
case where we get setclientid's with the same client-provided id.
There'd be no change of behavior in the case of multiple clients sharing
an IP (which is fine, of course).

> > For AUTH_SYS, all the FreeBSD server does is expect the same uid#.

Yeah, but that's probably usually the same between clients.

--b.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 19 Mar 2012 13:47:14 -0400

> an IP (which is fine, of course).

The migration draft proposes that clients use the same nfs_client_id4 string for all of a server's IP addresses.  Would a server then be obliged to return NFS4ERR_CLID_IN_USE if a client attempts a SETCLIENTID with the same boot verifier and nfs_client_id4 on more than one IP address for the same server?

IMO the server should not try to sort this situation out.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: "Myklebust, Trond" <[email-masked]>
Date: Mon, 19 Mar 2012 18:24:28 +0000

> The migration draft proposes that clients use the same nfs_client_id4 string for all of a server's IP addresses.  Would a server then be obliged to return NFS4ERR_CLID_IN_USE if a client attempts a…

Only if the client uses the wrong principal.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: "Myklebust, Trond" <[email-masked]>
Date: Mon, 19 Mar 2012 18:26:22 +0000

> have left to go on when the client's identifier-generation is messed up

Yes, but IP addresses can be reassigned dynamically. That's one of the
reasons for wanting a client id in the first place...

NetApp

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 19 Mar 2012 14:27:12 -0400

> The migration draft proposes that clients use the same nfs_client_id4 string for all of a server's IP addresses.  Would a server then be obliged to return NFS4ERR_CLID_IN_USE if a client attempts a…

That's also not this case, sorry, this time with all the conditions:

	- if the nfs_client_id4 is the same, and
	- if the flavor is auth_sys, and
	- if the client IP address is different,
	- then return NFS4ERR_INUSE.

--b.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 19 Mar 2012 14:29:46 -0400

> 	- then return NFS4ERR_INUSE.

This still breaks for multi-homed servers and UCS clients.  The client IP address can be different depending on what server IP address the client is accessing, but all the other parameters are the same.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 19 Mar 2012 14:39:55 -0400

> This still breaks for multi-homed servers and UCS clients.  The client IP address can be different depending on what server IP address the client is accessing, but all the other parameters are the…

OK.  So probably there's nothing we can do to help here.

As a bandaid maybe a rate-limited log message ("clientid X now in use
from IP Y") might help debug these things....

--b.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 19 Mar 2012 14:42:30 -0400

> from IP Y") might help debug these things....

Hm, OK.  That implies your server implementation assumes that a clientid4 maps to exactly one client IP address at a time.

---

From: Nikolaus Rath <[email-masked]>
Date: Mon, 19 Mar 2012 14:43:47 -0400

> from IP Y") might help debug these things....

Since you guys keep Cc'ing me, I'll chime in with a rather naive
suggestion: if all that's required is a unique id for every client, why
not use the MAC of the first network interface, independent of it being
used for communication with the server?

Best,

   -Nikolaus

-- 
 »Time flies like an arrow, fruit flies like a Banana.«

  PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6  02CF A9AD B7F8 AE4E 425C

---

From: Nikolaus Rath <[email-masked]>
Date: Mon, 19 Mar 2012 14:51:02 -0400

"J. Bruce Fields" <[email-masked]> writes:
>>>>>> >> > > > > IMO, the server should do a comparison of the nfs_client_id4
>>>>>> >> > > > > strings, and nothing else.
>>>>> >> > > >
>>>>> >> > > > We're supposed to return CLID_INUSE when we see a setclientid
>>>>> >> > > > from a "different" client using the same string, to keep
>>>>> >> > > > clients from doing mischief with other clients' state (either
>>>>> >> > > > maliciously or, as in this case, accidentally).
>>>>> >> > > >
>>>>> >> > > > "Different" here is defined as "not having the same principal".
>>>>> >> > > > I know what that means in the krb5 case, but I'm less certain
>>>>> >> > > > in the auth_sys case.
>>>> >> > >
>>>> >> > > setclientid's to come from the same client IP address at least in
>>>> >> > > the auth_sys case, or could that break multi-homed clients?
>>>> >> > >
>>> >> > I think that even a dhcp lease renewal might result in a different
>>> >> > client IP, if the client has been partitioned from the dhcp server
>>> >> > for a while.
> >
> > Yeah, but by that point the client's v4 lease is probably expired
> > anyway so the client's not likely to be bothered by the NFS4ERR_INUSE.
> >
>>> >> > I'm not convinced that different client IP# implies different
>>> >> > client. (Even "same ip# implies same client" might not be true, if
>>> >> > the dhcp server assigned the IP# to another machine while the
>>> >> > client was partitioned from the dhcp server, I think? I haven't
>>> >> > looked at current dhcp implementations, but it seems conceivable to
>>> >> > me.)
>>> >> > 
>> >> Oh, and what about the case of 2 clients that are sitting behind the
>> >> same NAT gateway? (I think they'd both be seen as having the client
>> >> host ip# of the gateway, but with different TCP connections on
>> >> different client port#s.)
> >

At least in the case that sparked this discussion, it would already be
enough to return NFS4ERR_INUSE only if the client id is being reassigned
*and* has a 0.0.0.0 (aka autodetection failed) value.

Best,

   -Nikolaus

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 19 Mar 2012 14:54:21 -0400

> Hm, OK.  That implies your server implementation assumes that a clientid4 maps to exactly one client IP address at a time.

OK, agreed.  So how about something like "state for client X previously
established from IP Y now cleared from IP Z"  ??

(Assuming it's only the I-just-rebooted setclientid case that's likely
to be the sign of a problem.)

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 19 Mar 2012 14:56:55 -0400

> *and* has a 0.0.0.0 (aka autodetection failed) value.

The clientid's supposed to be totally opaque to the server, so
recognizing that case (even knowing where in the clientid to look for
the "0.0.0.0") would require the server to behave in a way that was
extremely specific to particular versions of the Linux client--something
we definitely want to avoid doing.

--b.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 19 Mar 2012 15:00:57 -0400

> to be the sign of a problem.)

We would see that only in the case where the boot verifier and the client IP change at the same time.  That can happen legitimately, too, if the client has a dynamically assigned IP address.  Maybe this event is only interesting if it happens more than once during the same second.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 19 Mar 2012 15:08:53 -0400

> We would see that only in the case where the boot verifier and the client IP change at the same time.  That can happen legitimately, too, if the client has a dynamically assigned IP address.  Maybe…

"Warning: IP addresses Y and Z currently appear to be in a bitter
struggle over client id X"....

That may be getting complicated enough to not be worth it except as a
part of some more general statistics.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Mon, 19 Mar 2012 18:25:33 -0400 (EDT)

Nikolaus Rath wrote:
> >>>> The migration draft proposes that clients use the same
> >>>> nfs_client_id4 string for all of a server's IP addresses. Would a
> >>>> server then be obliged to return NFS4ERR_CLID_IN_USE if a client
> >>>> attempts a SETCLIENTID with the same boot verifier and
> >>>> nfs_client_id4 on more than one IP address for the same server?
> >>>
> >> This still breaks for multi-homed servers and UCS clients. The
> >> client IP address can be different depending on what server IP
> >> address the client is accessing, but all the other parameters are
> >> the same.
> >
I think this works fairly well for "real hardware", but I'm not so sure
about clients running in VMs. (I don't really know how the VMs assign
MAC addresses to their fake net interfaces and what uniqueness guarantees
those have. I remember the old freebie VMware client for Windows just
had a config file that assigned the MAC. I bet half the installations
on the planet had the same MAC as the default config file:-)

rick
ps: Also, although it's not very relevant, getting the MAC address of
    the first ethernet interface isn't easy in FreeBSD. I have no idea
    if the same is true of Linux. (I'd also be worried that "first"
    might not be fixed?)

---

From: Rick Macklem <[email-masked]>
Date: Mon, 19 Mar 2012 18:31:32 -0400 (EDT)

Chuck Lever wrote:
> >>>>> from
> >>>>> a
> >>>>> I
> >>>>> know
> >>>> the
> >>> client
> > anyway
> >>> client.
> The migration draft proposes that clients use the same nfs_client_id4
> string for all of a server's IP addresses. Would a server then be
> obliged to return NFS4ERR_CLID_IN_USE if a client attempts a
> SETCLIENTID with the same boot verifier and nfs_client_id4 on more
> than one IP address for the same server?
> 
0 maybe;-) But that's all the RFC says, so I think all a server can
do is hope the clients do a good just of generating unique
nfs_client_id4 strings?

rick

---

From: Nikolaus Rath <[email-masked]>
Date: Tue, 20 Mar 2012 09:29:57 -0400

> on the planet had the same MAC as the default config file:-)

But as I understand, the clientid doesn't have to globally unique, just
unique for the given NFS server. I think if you have two virtual
machines with the same MAC connecting to the same NFS server, you have
different problems anyway.

>     if the same is true of Linux. (I'd also be worried that "first"

It doesn't need to be the same interface all the time, I just meant the
first as in not a specific one.

Best,

   -Nikolaus

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 20 Mar 2012 13:55:52 +0000

> first as in not a specific one.

Yes it does have to be the same interface all the time. Otherwise the
server cannot tell that this is the same client booting up again.

...and yes, this is a problem not only on FreeBSD, but for Linux too.
The fact that devices can come up (and go down too in the case of
hotplugging) in any order is the main reason why the ethernet device
naming scheme was changed recently.

Trond

NetApp

---

From: Chuck Lever <[email-masked]>
Date: Tue, 20 Mar 2012 10:01:41 -0400

Sent from my iPad

On Mar 20, 2012, at 9:29 AM, Nikolaus Rath <[email-masked]> wrote:

> unique for the given NFS server.

Global uniqueness is required to support Transparent State Migration.

> I think if you have two virtual

The nfs_client_id4 string must not change across a reboot.  Otherwise the server won't be able to figure out which open and lock state to discard when the client reboots.

> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to [email-masked]
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

---

From: Nikolaus Rath <[email-masked]>
Date: Tue, 20 Mar 2012 10:36:30 -0400

> server cannot tell that this is the same client booting up again.

The likelihood of the interface order changing is certainly much lower
than the likelihood of the IP address changing (which I understand is
currently used for the clientid), so the situation would still improve.

Best,

   -Nikolaus

---

From: Nikolaus Rath <[email-masked]>
Date: Tue, 20 Mar 2012 10:38:02 -0400

> The nfs_client_id4 string must not change across a reboot.  Otherwise the server won't be able to figure out which open and lock state to discard when the client reboots.

But I thought that at the moment one of the client's ips is used for the
clientid? That's certainly changes regurlarly for DHCP or multi-homed
clients.

Best,

   -Nikolaus

---

From: Chuck Lever <[email-masked]>
Date: Tue, 20 Mar 2012 11:53:31 -0400

> clientid? That's certainly changes regurlarly for DHCP or multi-homed

You are correct that using an IP address in this string is not optimal.  Obtaining some other unique identifier (say a MAC address) that won't ever change across a reboot is a difficult challenge.  Currently most client implementations do insert both the client IP and server IP address in the nfs_client_id4 string.  In fact, RFC 3530 recommends the use of IP addresses as part of this string.

However, NFSv4.1 I believe is driving the adoption of uniform client strings (that is, the nfs_client_id4 string is the same no matter what server the client talks to), and Transparent State Migration support in NFSv4.0 may well also require the use of a uniform client string.

In other words, I think the use of IP addresses in the client strings will be phased out over time.  We believe that using a MAC address is not much better than an IP address.  I've got a patch for Linux that allows administrators to insert a UUID into this string, set via a kernel boot parameter (much like the root= boot parameter).  This is optional, and can remain fixed over the life of the client system.  Aside from using the client's FQDN, a UUID is probably the best we can do.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 20 Mar 2012 16:49:34 +0000

> currently used for the clientid), so the situation would still improve.

It is still easy to find examples of systems for which this breaks:

What do you do for boxes that only have an infiniband connection, for
instance? AFAIK, a lot of IPoIB implmentations don't have reboot-safe
MAC addresses.
Then there is the fact that some of the cheaper NICs don't have
permanent MAC addresses. The MAC address usually gets set as part of the
boot process.

Finally, there is the issue that the NIC is a globally shared hardware
component. If I am running virtual machines in containerised
environments (and yes, I am about to merge the NFS support for
containers into Linux-3.4) then having the NFS clients in all the
different containers identify themselves using the same MAC address will
to lead to some serious brokenness.

NetApp