Subject: [nfsv4] 4.0 trunking

From: [email-masked] (J. Bruce Fields)
Date: Wed, 7 Sep 2016 17:20:39 -0400

In
https://tools.ietf.org/html/draft-ietf-nfsv4-migration-issues-10#section-5.4.2

	In the face of possible trunking of server IP addresses, the
	client will use the receipt of the same clientid4 from multiple
	IP-addresses, as an indication that the two IP- addresses may be
	trunked and proceed to determine, from the observed server
	behavior whether the two addresses are in fact trunked.

But I can't find any further discussion of how a client might make that
determination.  Am I overlooking it?

(It appears that the Linux client is trying to do that by sending a
setclientid_confirm to server1 using the (clientid,verifier) returned
from a setclientid reply from server2.  That doesn't look correct to
me.)

--b.

---

From: David Noveck <[email-masked]>
Date: Wed, 7 Sep 2016 20:14:58 -0400

> determination.  Am I overlooking it?

It's actually in section 5.8 of RFC7931.

I've only been keeping draft-ietf-nfsv4-migration-issues alive because of
the section dealing with issues relating to v4.1.  Otherwise, I would have
let the thing expire.  The next time I update this, I'll probably collapse
sections 4 and 5 to a short section saying that all the
v4.0 issues were addressed by publication of RFC7931.

> (It appears that the Linux client is trying to do that by sending
> a setclientid_confirm to server1 using the (clientid,verifier)
> returned from a setclientid reply from server2.  That doesn't look
> correct to me.)

It seems kind of weird but the idea is that if you get the same clientid
from two server IP address they are probably connected to the same server
(i.e are trunked), but there is a chance that having the same clientid is a
coincidence,  The idea is that if these are the same server using the
verifier will work but if they are different servers it won't work but will
be harmless.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 7 Sep 2016 21:05:32 -0400

> It's actually in section 5.8 of RFC7931.

Oops, thanks!

Looking at that, I think none of this is true:

	Note that the NFSv4.0 specification requires the server to make
	sure that such verifiers are very unlikely to be regenerated.

Verifiers given out by one server shouldn't be reused, sure, but that's
quite different from the claim that collisions *between* servers are
unlikely.

	Given that it is already highly unlikely that the clientid4 XC
	is duplicated by distinct servers,

Why is that highly unlikely?

	the probability that SCn is
	duplicated as well has to be considered vanishingly small.

There's no reason to believe the probability of a verifier collision is
uncorrelated with the probability of a clientid collision.

The Linux server is generating clientid as a (boot time, counter) pair.
So collision between servers started at the same time (probably not that
unusual) is possible.

Ditto for the verifier--it's a (boot time, counter) pair, with the
second part coming from a different counter, but normally I think
they'll get incremented at the same time, so clientid and verifier are
highly correlated.

So, we can mitigate this by adding some randomness, OK.  But that's a
new requirement not apparent from 3530, and I wouldn't be surprised if
other servers have similar issues.

--b.

> the section dealing with issues relating to v4.1.Otherwise, I would have

---

From: David Noveck <[email-masked]>
Date: Sat, 10 Sep 2016 15:38:29 -0400

Thanks for pointing this out.

Before I address the details of this, let me state my overall position:

   - I think it is correct that RFC7931 overstates the degree of assurance
   that one might reasonably have regarding the unlikelihood of spurious
   collision of clientid4's and verifiers.
   - Nevertheless, I don't think the situation is as bleak as you paint it,
   with regard to the Linux server approach to these matters that you
   describe.  This is basically because I don't see the issue of the
   correlation between clientid4's and verifiers the same way that you do. See
   below.
   - I think it is possible to address the issue satisfactorily in the
   context of an errata rfc7931 errata and will start working on that.
   - Given the weakness of the rfc 3530/7530 requirements in this area, we
   may need (yet another) RFC updating 7530 at some point.
   - I see that as a longer-term effort, since the practices that you
   describe will not result in large numbers of spurious collisions and
   clients can adapt the algorithm to require additional verification.
   - If there are servers out there whose practices are significantly more
   troublesome than the ones you describe, we need to find out soon.  Given
   that many of those responsible for v4.0 implementations may not be reading
   this list, I suggest we discuss this at the October Bakeathon.

> The Linux server is generating clientid as a (boot time, counter) pair.

I'm assuming the boot time is in the form of a 32-bit unsigned number of
seconds after some fixed date/time.  Correct?

If that is the case, duplicates would require that:
1. Two servers be booted during the same second (e.g. when power came back
on after a disruption).
2. Each of the servers has received the same number of client SETCLIENTIDs
(mod 4 billion).

Clearly this is possible although I would say that it is unlikely.
Nevertheless, the "highly unlikely" in RFC7931 is overstating it.

The case that would be more worrisome would be one which a server simply
used a 64-bit word of byte-addressable persistent memory (e.g. NVDIMM, 3d
xpoint) to maintain a permanent counter, dispensing with the boot time.
That is allowable as RFC7530 stands.  I don't expect that this is a current
problem but, given where persistent memory is going (i.e. cheaper and more
common), we could see this in the future.

Clearly, spurious clientid4 collisions are possible, which is the point of
the whole algorithm.

> Ditto for the verifier--it's a (boot time, counter) pair,

I presume the same boot time format is used.  Correct?

>with the second part coming from a different counter, but
> normally I think they'll get incremented at the same time,

It is true that they will both get incremented in many common (i.e.
"normal") situations.

However, this common case is not the only case.

In particular, in the case in which one or more clients are following the
guidance in RFC7931 and there are trunked server addresses, you will have
occasions in which the verifier counter will get incremented and the
clientid4 counter will not.
1.  If the SETCLIENTID is used to modify the callback information
2.  If two different SETCLIENTID operations are done by the same client

>so clientid and verifier are highly correlated.

I don't think so.  The boot time portions are clearly the same, but given
the likely occurrence of cases 1. and 2.these counters will typically have
different values.  They may be correlated in the sense that the difference
between the two values is likely not to be large, but that does not mean
that the probability of them having the same value is necessarily substantial.
Once there are any occasions in which the verifiers are incremented and the
clientid's are not, these values are essentially uncorrelated.  The only
way that they can collide at the same time the clientids collide is when
the number of instances of the cases 1. and 2. above is a multiple of
2^32.  While I think that RFC7931's "vanishingly small" is not correct, I
would argue that this brings us into "highly unlikely" territory. Also, the
existing text does suggest that you can repeat the procedure any number of
times to reduce the likelihood of a spurious determination that two
addresses are trunked.

> So, we can mitigate this by adding some randomness, OK.

One version of that might be to start the counter fields with the
nanoseconds portion of the boot time.

Alternatively, you might  keep the  current counter each starting at zero
but take advantage of the fact that clientid and verfier are used
together.  To do that the timestamp, in clientid4 might be boot time in
seconds while the one in the verifier would be boot time nanoseconds within
the second during which the boot occurred. That would reduce the frequency
of verifier collisions but this might not be necessary, if, as i expect,
the algorithm will usually result in distinct verifiers anyway.

> But that's a new requirement not apparent from 3530,

I think the 7931 text can be patched up via an errata, but if not we will
have to consider how and whether to address the issue in 7530.
Unfortunately, that would be kind of big for an errata.

If no existing servers have a substantial issue, then we do have the option
of doing a longer-term RFC updating 7530.  This could be directed to the
(probably now) hypothetical case of a server not using boot time and
 maintaining 64-bit global persistent counts.

Such servers would have a higher probability of clientid4 conflict and
verifier conflict than the ones you mention, while staying within the
rfc7530 requirements.

> and I wouldn't be surprised if other servers have similar issues.

Those who read this list have been notified of the issue by your message.

The problem is with server implementers who don't read this list.  In
theory, errata should take care of that, but aside from the difficulty of
arriving at a suitable change and getting it through, it may be that many
v4.0 implementers and maintainers may not be reading this list or paying
much attention to errata either.  Perhaps we can discuss this in Westford
where many implementers, who might not be all that aware of stuff on the
working group list, would be present.  That's also one way to see if there
are existing servers where this is big problem.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Sat, 10 Sep 2016 16:03:55 -0400

> 1. Two servers be booted during the same second (e.g. when power came back

Right, or routine maintenance, or whatever.

> Nevertheless, the "highly unlikely" in RFC7931 is overstating it.

It turns out that if you have a hundred servers that get rebooted
simultaneously for a kernel update or some similar routine maintenance,
this happen every time.  (I'm a step removed from the original case
here, but I believe this is more-or-less accurate and not hypothetical.)

Pretty sure you can easily hit this without that big a setup, too.

> xpoint) to maintain a permanent counter, dispensing with the boot time.

That'd be something worth looking into in cases the users have the right
hardware (not always).  The workaround I'm going with for now is
initializing the counter part to something random.  (Random numbers may
not be completely reliable at boot time either, but I suspect they'll be
good enough here...).

> 2.  If two different SETCLIENTID operations are done by the same client

Sure.

> 2^32.

I'm not following.  I think you may be confusing clientid-verifier
collisions with verifier-verifier (from different server) collisions.
The latter is what matters.

The confusion may be my fault.  I should have said something like:
chance of a collision between two clientid's is correlated with chance
of a collision between two verifiers given out by the same two servers.
So adding in a verifier comparison doesn't decrease the probability as
much as you'd expect.

> While I think that RFC7931's "vanishingly small" is not correct, I
> would argue that this brings us into "highly unlikely" territory.

So, unfortunately, we appear to have an actual real-life case where this
happens all the time.

> Also, the

My intuition is that that would mean a lot of effort for a disappointing
reduction in the probability of a bug, but I haven't done any
experiments or thought this through much.

--b.

> are existing servers where this is big problem.

OK!

Assuming nobody's doing anything very complicated--it's probably also
not difficult to guess algorithms in testing.  Something we can do in
Westford, or maybe before if anyone has access to a variety of servers.

--b.

---

From: "Adamson, Andy" <[email-masked]>
Date: Mon, 12 Sep 2016 12:59:05 +0000

IMHO we should not worry too much about NFSv4.0 trunking as NFSv4.1+ solves this issue. Trunking is simply one more reason to move from NFSv4.0 to NFSv4.1.

—>Andy

> On Sep 10, 2016, at 4:03 PM, J. Bruce Fields <[email-masked]> wrote:
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 14 Sep 2016 11:48:54 -0400

On Sat, Sep 10, 2016 at 04:03:55PM -0400, J. Bruce Fields wrote:
> hardware (not always).

(Though note you'd still want to randomize the starting value.)

> The workaround I'm going with for now is

> > > Ditto for the verifier--it's a (boot time, counter) pair,

Sorry, I got this wrong, our verifier is (setclientid time, counter).

The problem remains, because as with boot times, setclientid times are
often clustered together.  (In the case of a client boot or server
reboot, the client is likely sending setclientid's to all the servers it
cares about at the same time.)

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 20 Sep 2016 09:28:23 -0400

On Sat, Sep 10, 2016 at 03:38:29PM -0400, David Noveck wrote:
> - If there are servers out there whose practices are significantly
> more troublesome than the ones you describe, we need to find out soon.
> Given that many of those responsible for v4.0 implementations may not
> be reading this list, I suggest we discuss this at the October
> Bakeathon.

By the way, that'll be more useful the more server implementations are
represented.  Last I heard I think we have Linux, Primary Data, and
Solaris, and I'm not sure about anyone else (Ontap? EMC? Windows?).

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 20 Sep 2016 17:39:31 -0400

> IMHO we should not worry too much about NFSv4.0 trunking as NFSv4.1+ solves this issue. Trunking is simply one more reason to move from NFSv4.0 to NFSv4.1.

I tend to agree, but since it got implemented an was causing a real bug,
it was hard to ignore....

After a patch (55b9df93ddd6 "NFSv4/v4.1: Verify the client owner id
during trunking detection", if anyone cares) the damage seems restricted
to the non-default "migration" case, making this less of a concern.

I wonder if there's an easy fix, though: if I'm reading rfc 7931 right,
it works (slightly simplified) like this:

	1. a SETCLIENTID to IP address X1 returns a clientid.  You
	confirm it and carry on.

	2.  Some time later a SETCLIENTID to IP address X2 returns the
	same clientid.  To check whether X1 and X2 actually point to the
	same server:

	3. Send a callback-changing SETCLIENTID to X1.  Get back
	(clientid, verifer).

	4. Confirm it with a SETCLIENTID_CONFIRM to X2.

If the SETCLIENTID_CONFIRM succeeds then they're the same server.
Servers are required to return an error (STALE_CLIENTID, I think?) if
the verifier in the SETCLIENTID_CONFIRM arguments doesn't match.

But, if the SETCLIENTID at step 2 just happened to return the same
verifier as the SETCLIENTID at step 3, then the SETCLIENTID_CONFIRM at
step 4 will succeed even if the two servers are different.  Hence the
bug.

The thing is, though, that in that case we already have our answer at
step 3: if X1 pointed to the same server as X2, then the SETCLIENTID to
X1 would have to return a distinct verifier.  We can stop there.

With that addition, don't we get the right answer every time, without
any unjustified assumptions about the randomness of server's clientid
and verifier generation?  Or am I missing something?

--b.

---

From: David Noveck <[email-masked]>
Date: Tue, 20 Sep 2016 20:55:31 -0400

>The thing is, though, that in that case we already have our
> answer at step 3:
> if X1 pointed to the same server as X2, then the
> SETCLIENTID to X1 would have to return a distinct verifier.

True but the converse doesn't hold.  If you get a distinct verifier,
it could be because X1 and X2 are the same server, but the other
possibility is that X1 and X2 are different servers and, if they are, they
may well happen to have different verifiers.

But I think you are on to something.  If the verifiers for X1 and X2 are V1
and V2 respectively, if V2 works successfully then you have to go back and
use V1 on X1.  If the SETCLIENTID_CONFIRM to X2 using V2 succeeds, you can
go back and try another confirm to X1 using V2.  If X1 and X2 are distinct,
that should still succeed.  On the other hand, if they are the same server
the creation of V2 should replace V1 and it will become invalid.

> We can stop there.

Or somewhere.

On Tue, Sep 20, 2016 at 5:39 PM, J. Bruce Fields <[email-masked]>
wrote:

> > IMHO we should not worry too much about NFSv4.0 trunking as NFSv4.1+
> solves this issue. Trunking is simply one more reason to move from NFSv4.0
> to NFSv4.1.
>
> more
> > >> On Wed, Sep 7, 2016 at 9:05 PM, J. Bruce Fields <[email-masked]
> >
> > >> wrote:
> > >>
> > >>>> I've only been keeping draft-ietf-nfsv4-migration-issues alive
> because
> > >>> of
> > >>>> (i.e are trunked), but there is a chance that having the same
> clientid
> > >>> is a
> > >>>> verifier will work but if they are different servers it won't work
> but
> > >>> will
> > >>>> be harmless.
> > >>>>
> > >>>>
> > >>>>>        IP-addresses, as an indication that the two IP- addresses
> may
> > >>> be

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 20 Sep 2016 22:45:31 -0400

On Tue, Sep 20, 2016 at 08:55:31PM -0400, David Noveck wrote:
> True but the converse doesn't hold.

Agreed, I was suggesting stopping here only in the case the verifiers
match, and continuing otherwise.

> If you get a distinct verifier,

Yes.

> the creation of V2 should replace V1 and it will become invalid.

I think that would work, though what I was suggesting was a few less
steps, roughly:

	 2.  Some time later a SETCLIENTID to IP address X2 returns the
	 same clientid, and verifier V2.  To check whether X1 and X2
	 actually point to the same server:

	 3. Send a callback-changing SETCLIENTID to X1.  Get back
	 (clientid, V1).  If V1 == V2, then X1 and X2 are different
	 servers.  Otherwise:

	 4. Send a SETCLIENTID_CONFIRM(clientid, V1) to X2.  If it
	 succeeds, they're the same, otherwise, they're different.

--b.

---

From: David Noveck <[email-masked]>
Date: Wed, 21 Sep 2016 05:09:59 -0400

I think that works.  I'm going to look at how to update section 5.8 of
RFC7931.

On Tue, Sep 20, 2016 at 10:45 PM, J. Bruce Fields <[email-masked]>
wrote:

> > > > > On Sep 10, 2016, at 4:03 PM, J. Bruce Fields <[email-masked]
> >
> > > wrote:
> > > > >
> > > > >>   correlation between clientid4's and verifiers the same way that
> you
> > > do. See
> > > > >>   below.
> > > > >>   troublesome than the ones you describe, we need to find out
> soon.
> > > Given
> > > > >>   that many of those responsible for v4.0 implementations may not
> be
> > > reading
> > > > >> I'm assuming the boot time is in the form of a 32-bit unsigned
> number
> > > of
> > > > >> 1. Two servers be booted during the same second (e.g. when power
> came
> > > back
> > > > >> on after a disruption).
> > > > >
> > > > > not be completely reliable at boot time either, but I suspect
> they'll
> > > be
> > > > > good enough here...).
> > > > >
> > > > >> That is allowable as RFC7530 stands.  I don't expect that this is
> a
> > > current
> > > > >> problem but, given where persistent memory is going (i.e. cheaper
> and
> > > more
> > > > >> In particular, in the case in which one or more clients are
> following
> > > the
> > > > >> guidance in RFC7931 and there are trunked server addresses, you
> will
> > > have
> > > > >> I don't think so.  The boot time portions are clearly the same,
> but
> > > given
> > > > >> the likely occurrence of cases 1. and 2.these counters will
> typically
> > > have
> > > > >> between the two values is likely not to be large, but that does
> not
> > > mean
> > > > >> Once there are any occasions in which the verifiers are
> incremented
> > > and the
> > > > >> clientid's are not, these values are essentially uncorrelated.
> The
> > > only
> > > > >> way that they can collide at the same time the clientids collide
> is
> > > when
> > > > >> Alternatively, you might  keep the  current counter each starting
> at
> > > zero
> > > > >> together.  To do that the timestamp, in clientid4 might be boot
> time
> > > in
> > > > >> seconds while the one in the verifier would be boot time
> nanoseconds
> > > within
> > > > >> I think the 7931 text can be patched up via an errata, but if not
> we
> > > will
> > > > >> If no existing servers have a substantial issue, then we do have
> the
> > > option
> > > > >> of doing a longer-term RFC updating 7530.  This could be directed
> to
> > > the
> > > > >> arriving at a suitable change and getting it through, it may be
> that
> > > many
> > > > >> where many implementers, who might not be all that aware of stuff
> on
> > > the
> > > > >> working group list, would be present.  That's also one way to see
> if
> > > there
> > > > >>>>> But I can't find any further discussion of how a client might
> make
> > > that
> > > > >>> There's no reason to believe the probability of a verifier
> collision
> > > is
> > > > >>> So collision between servers started at the same time (probably
> not
> > > that
> > > > >>> unusual) is possible.
> > > > >>>
> > > > >>> they'll get incremented at the same time, so clientid and
> verifier
> > > are
> > > > >>> highly correlated.
> > > > >>>
> > > > >>> new requirement not apparent from 3530, and I wouldn't be
> surprised
> > > if
> > > > >>>> the section dealing with issues relating to v4.1.Otherwise, I
> would
> > > have
> > > > >>>> let the thing expire.  The next time I update this, I'll
> probably
> > > > >>> collapse
> > > > >>>> from two server IP address they are probably connected to the
> same
> > > server
> > > > >>>> coincidence,  The idea is that if these are the same server
> using
> > > the
> > > > >>>>> But I can't find any further discussion of how a client might
> make
> > > that
> > > > >>>>> from a setclientid reply from server2.  That doesn't look
> correct
> > > to
> > > > >>>>> me.)
> > > > >>>>>
> > > > >>>>> --b.
> > > > >>>>>
> > > > >>>
> > > > >

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 21 Sep 2016 10:14:09 -0400

> I think that works.  I'm going to look at how to update section 5.8 of

Great, let me know when there's something to review.

The Linux client looks like it's actually doing something even simpler,
basically:

	3. Confirm that SETCLIENTID with SETCLIENTID_CONFIRM(client, V2)
	sent to X1.

And I don't see how to shoehorn this extra check in there, so I think it
needs to add in the extra callback-changing SETCLIENTID to X1.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 21 Sep 2016 15:53:39 -0400

> needs to add in the extra callback-changing SETCLIENTID to X1.

Oh, now that I look at it I think it's really easy.  All we have to do
is check that the verifier has changed since step 1.  If it has,
continue to step 3, if it hasn't, they're different servers.

--b.

---

From: David Noveck <[email-masked]>
Date: Thu, 22 Sep 2016 07:59:40 -0400

> Great, let me know when there's something to review.

Right now I'm having trouble deciding exactly what needs to be fixed.
Clearly, the paragraph in section 5.8 containing the phrase "vanishingly
small" needs to be corrected since it is, as you pointed out, incorrect.
In the future, I'll refer to this as "the VS paragraph".

The question is what else, if anything needs to be fixed.  I had been
thinking that the problems you were seeing indicated that the algorithm in
section 5.8 needed correcting.  As it now appears that the Linux client had
implemented something different than that, the correctness of the algorithm
in section 5.8 has to be considered an open question.

> The Linux client looks like it's actually doing something even simpler,

And thus different!

So the questions we need to address are:

   - Whether the algorithm is section 5.8 has been implemented by any
   client.
   - Whether anybody has any reason to believe that that algorithm in
   section 5.8 would not work as described there, even when servers do not
   conform to the VS paragraph.

>       3. Confirm that SETCLIENTID with SETCLIENTID_CONFIRM(client, V2)

I don't see how that would be helpful.  By hypothesis, X1 and X2 both think
that a specific clientid4/verifier pair is valid.  If that is the case, a
SETCLIENTID_CONFIRM to X1 will succeed, independently of whether X1 and X2
are the same server or not.

> And I don't see how to shoehorn this extra check in there,

So, is the non-shoehorn-able extra check you are referring to the list item
3 above?  If that is the case, then it is easy to see how, given what you
have told us about how the server is assigning clientid4 and verifier
values, why this is commonly resulting in identifying separate servers as
the same.  On the other hand, I can see how someone reading the VS
paragraph might assume that the probability of this happening with two
different servers was "vanishingly small" and so punt.

> so I think it

Although the text is not as clear as it might be , the intention of the
algorithm in section 5.8 is that the SETCLIENTID be done to X2 and
confirmed on X1.  It probably would work the other way around but the
critical part is that the SETCLIENTID and SETCLIENTID_CONFIRM have to go to
different IP addresses, in order to help determine whether  those two
addresses are connected to the same server.

> Oh, now that I look at it I think it's really easy.

Unfortunately, my experience has been that the probability, in this area,
of thinking that something is really easy, and it turning out tht way, is
"vanishingly small" :-(

> All we have to do
> is check that the verifier has changed since step 1.

How exactly do you expect to do that?

>If it has, continue to step 3,

If it has, then I don't see how they could be different servers.

> if it hasn't, they're different servers.

I'm not sure of the exact context here, but you have to be aware of the
possibility that a new verifier will not be generated because a given
request is treated as a replay.  If you are not doing a callback-changing
SETCLIENTIDs, it is easy to fall into this case.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 22 Sep 2016 08:51:45 -0400

> in section 5.8 has to be considered an open question.

It has the same problem; I could propose some language if you'd like.

> SETCLIENTIDs, it is easy to fall into this case.

7530 requires a new verifier even in the case the callback information
isn't new.  ("The server treats this as a probable callback information
update and records an unconfirmed { v, x, c, k, t } and leaves the
confirmed { v, x, c, l, s } in place, such that t != s.  It does not
matter whether k equals l or not.")

If you think there's a risk anyway for some reason (bugs, or xid
wraparound, maybe?), then we could continue advising a callback-changing
SETCLIENTID here.

(The Linux client always changes the callback information on this
setclientid so my particular case isn't at issue.)

To further muddy the waters: I also found Linux knfsd isn't always
rejecting SETCLIENTID_CONFIRMs with incorrect verifiers when it should.
Even with that bug fixed, this problem is still likely thanks to the
weak verifier generation (which I'm also fixing).  So I'm still
motivated to fix the protocol.

I checked our revision history, and that Linux nfsd bug has been there
undetected from the start (13-14 years).  There's some risk to depending
on behavior that implementations have never previously depended on.

--b.

---

From: David Noveck <[email-masked]>
Date: Thu, 22 Sep 2016 10:46:35 -0400

> > As it now appears that the Linux client had
> > implemented something different than that, the correctness of > > the
algorithm

> It has the same problem;

Please explain what the problem is.  If we don't agree on what the problem
is, it is hard to come up with a solution.

One way to do this would be to give a case where a
client implementing section 5.8 might interact with two servers and could
arrive at the wrong result, assuming the servers meet the requirements of
RFC7530, which are, as you pointed out, considerably weaker than those in
the VS paragraph in section 5.8.

> I could propose some language if you'd like.

If you are talking about language proposing how to address the problem, you
feel exists, then you could do that.  However I feel it would be better if
we focused first on the question of what the problem is.

> 7530 requires a new verifier even in the case the callback
> information isn't new.

Good point but I'm unclear about whether this would count as a verifier
"change".  It was never clear how you would test this.  Anyway, this is a
case in which a new verifier is generated but the old one remains valid
Whether this counts as the verifier being changed depends on how you test
for verifiers being changed.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 22 Sep 2016 12:04:45 -0400

> the VS paragraph in section 5.8.

I'll stick a variation on the shorthands I've been using but I beleive
I'm following section 5.8 otherwise:

1. A previous SETCLIENTID to X1 returned (clientid, verifier) (C, V1).
It was confirmed and kept renewed since.

2. A new SETCLIENTID to X2 returns (C, V2).

3. Confirm it with a SETCLIENTID_CONFIRM(C, V2) to X2.

4. Send a callback-updating SETCLIENTID(C) to X1, which returns (C, V1').

5. Send a SETCLIENTID_CONFIRM(C, V1') to X2.

Then consider X1 and X2 the same server iff 5 succeeds.

This can incorrectly identify X1 and X2 as the same if V1' happens to
equal V2.  In that case X2 will see the SETCLIENTID_CONFIRM as a replay
of the one in step 3.

The RFC is aware of the possibility but expects it to be rare because
clientids and confirm verifiers are both 64-bit values, and if they were
each chosen uniformly at random from that space, the chance of both
colliding would be insignificant.

In practice they may be chosen in a predictable fashion from a limited
part of the 64-bit space.  And the birthday paradox compounds this
problem in the presence of multiple servers.

Do we agree on the problem?

--b.

---

From: David Noveck <[email-masked]>
Date: Thu, 22 Sep 2016 19:50:44 -0400

> I'll stick a variation on the shorthands I've been using but I
> beleive I'm following section 5.8 otherwise:

> This can incorrectly identify X1 and X2 as the same if V1'
> happens to equal V2.

So section 5.8 is incorrect in telling you to do that.  Point taken.  Later
we'll discuss what to do instead.

> In that case X2 will see the SETCLIENTID_CONFIRM as a
> replay of the one in step 3.

True.

>The RFC is aware of the possibility but expects it to be rare
> because clientids and confirm verifiers are both 64-bit values,
> and if they were each chosen uniformly at random from that
> space, the chance of both colliding would be insignificant.

I don't know if I was assuming uniform random choice.  in any case, I
grossly underestimated the probability of collision.

> In practice they may be chosen in a predictable fashion from a
> limited part of the 64-bit space.  And the birthday paradox
> compounds this problem in the presence of multiple servers.

So now we want to look at how to correct the problem.  The last sentence of
that paragraph says "Note also that the callback update procedure can be
repeated multiple times to reduce the probability of further spurious
matches".   Given what we've seen about the probability of single
collisions, it might seem not worth doing additional iterations.  However,
if we continue your scenario, I think the probability of a false positive
can be reduced to exactly zero by doing just one more
SETCLIENTID/SETCLIENTID_CONFIRM
pair.

So now assume that after step 5, we assume X1 and X2 are distinct if the
operation fails.  However, if it succeeds, we do the following:

6. Send yet another callback-updating SETCLIENTID(C) to X1, which returns
(C, V1'').

Note that V1' != V1''

7. Send a SETCLIENTID_CONFIRM(C, V1'') to X2.

If this operation fails, we conclude that X1 and X2 are distinct and that
step 5 succeeded because V1' = v2.

If this operation succeeds, I believe we are justified in assuming that X1
and X2 are the same.  Suppose they are distinct.  In this case, step 7 can
only succeed if X2 received a SETCLIENTID returning C and V1''.  The only
SETCLIENTID X2 received that returned C had the verifier V2.  And V2 cannot
equal V1'', since, for us to be in this case V2 = V1' and V1 != V1''.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 23 Sep 2016 09:40:55 -0400

> equal V1'', since, for us to be in this case V2 = V1' and V1 != V1''.

I think that's correct.

What I expect we'll use on the Linux is a minor variation on its
existing algorithm:

	1. a SETCLIENTID to IP address X1 returns (C, V1).
 
	2.  Later a SETCLIENTID to IP address X2 returns (C, V2).  If
	V1==V2, the two servers are different, otherwise:

	3. Send a SETCLIENTID_CONFIRM(C, V2) to X1.

That final SETCLIENTID_CONFIRM succeeds if and only if X1 and X2 are the
same.

--b.

---

From: Andy Adamson <[email-masked]>
Date: Fri, 30 Sep 2016 16:25:00 -0400

Hi

Chuck Lever and I are writing the IETF spec describing the use of
fs_locations/fs_locations_info for client multipath discovery for
NFSv4.1 non-pNFS or MDS server trunking. Do you think we should bother
with the NFSv4.0 (clientID trunking) case?

-->Andy

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 30 Sep 2016 16:52:04 -0400

> with the NFSv4.0 (clientID trunking) case?

My feeling is that the problems we saw with 4.0 trunking are pretty easy
to fix.  So don't take my complaints there as a reason to rule it out.

Like you, though, I'd rather ignore it.

The reason to do it would probably be to deliver this to people stuck on
older OS's without 4.1 support, but those OS's are getting increasingly
difficult to hack additional features into anyway.

If it were just me, I'd do the 4.1 case first.  Then if it looked like
the same basic approach would cover 4.0 too--hey, why not.

--b.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 30 Sep 2016 16:55:16 -0400

> On Sep 30, 2016, at 4:52 PM, J. Bruce Fields <[email-masked]> wrote:
> 

I agree, 4.1 first, and 4.0 is worth looking into. Someone is bound to
read the trunking I-D and ask "why not 4.0" if we don't say something
about 4.0 in there. We don't have to spend a lot of time on it, but I
don't see why the RFC 7931 mechanism wouldn't permit clientid trunking
with 4.0.

--
Chuck Lever