Subject: Re: [nfsv4] Individual i-d on v4.0 migration issues

From: <[email-masked]>
Date: Thu, 1 Sep 2011 16:56:59 -0400

Some points that especially need to be clarified.

o You do not have to use the uniform model to get viable migration
  support.  Take a look at the results section to get some detail.

o Although it initially seemed that the problems that were seen were
  due to the non-uniform model, analysis showed that this was not the 
  case and that we needed to have lease merger working for both models.

o I just didn't think it was viable to tell those who followed the 
  recommended model, (even if it wasn't "RECOMMENDED"), to not support
  migration which is supposed to be a 4.0 feature and thus work in v4.0.

o If that isn't convincing, you might think about prototyping v4.0 migration
  support using the non-uniform model, following the i-d's (meta)-RECOMMENDATIONS.  
  There is expected to be a prototype server at the bthon supporting migration 
  in line with the i-d's approach.  You'll have to check with my co-authors 
  for details but my understanding is that it is expected to support clients 
  using either the uniform or the non-uniform model.

Of course this would make it even less appealing for you to choose the
uniform model, but nobody is telling you to or asking you to.  (or at least I'm
not and the i-d is not; can't speak for everybody).

I think I could write some text that fairly explains why you and 
others similarly situated might make the choice that you do, but forbidding 
everyone from choosing otherwise doesn't seem right to me.  For example,
your decision is based in part on the idea that you will have v4.1 migration 
support (reasonable choice for you) but I can't, in a v4.0-targeted spec, 
tell people to "wait for v4.1" to get a working version of a v4.0 feature.

Your suggestion about fs-locations on the rootfh is a good one.  I'm 
thinking that some version of it might be incorporated into the -01.
(although we will have some discussion about the appropriate SHOULD's and 
MUST's). The -01 should be out in time for the bthon (by which I mean 
36-48 hours before).

If we have productive discussion about the remaining issues at the bthon
and/or interoperability results to report on, that would make sense to put
in a -02.  That would have to come out some time after ietf82.

-----Original Message-----
From: Myklebust, Trond [mailto:[email-masked]] 
Sent: Wednesday, August 31, 2011 12:48 PM
To: Noveck, David
Cc: [email-masked]
Subject: RE: [nfsv4] Individual i-d on v4.0 migration issues

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, August 31, 2011 8:39 AM
> To: Myklebust, Trond
> Cc: [email-masked]
> Subject: RE: [nfsv4] Individual i-d on v4.0 migration issues
> 
> > While I agree with your argument, I don't think that it is
> sufficient,
> 
> What is it insufficient for?  It is intended to show that the only
> reason
> Given in RFC3530 and RFC3530bis-{1..13} is not  valid one. Since you
> agree
> with the argument, I take it you agree that it is sufficient for that
> purpose.
> 
> Do you think it is insufficient to cause us to go from "should" to
> "MAY"?
> My approach is different.  In the first place I don't think "should"
> makes
> sense for things like this, and if you don't like "MAY" you need to
> suggest
> some argument for "SHOULD" (use the non-uniform model) that is
> sufficient to
> make that case.  Now you have offered some sort of argument which if
> valid
> might demonstrate that (and I'll have some criticisms of it below),
but
> I
> think that the burden is on you to show that that uniform model cannot
> work and
> I don't think you understand how difficult that is to show.
> 
> When you say it "cannot work" you are saying that no algorithm can be
> devised
> to deal with this situation and that's why you need a real proof to
> show that
> since there are an infinite (although denumerable) set of possible
> algorithms.
> You have to show a situation that no algorithm can deal with and that
> is hard to do.
> 
> > The main problem with a client that is unaware of the fact that it
is
> > talking to the same server over 2 different IP addresses
> 
> He is unaware that he is talking to the same server but he has to be
> aware that he might be talking to he same server.
> 
> The current text tries to make it be clear that sometime he knows the
> servers
> Are different and sometimes he can find out they're the same, but some
> of
> the time he may not know for a while and has to be able to deal with
> that
> uncertain state.
> 
> > it may end up
> > performing conflicting stateful operations.
> 
> "conflicting" in what sense?  I don't think you've cited anything
> Below where the operations cancel each other out.
> 
> > then that client is going to find tracking the
> > seqids to be _very_ difficult, since it thinks it is working with 2
> > different seqid streams because it thinks it is dealing with 2
> different
> > servers.
> 
> We can get into a not very helpful dispute as you say "_very_
> difficult"
> and I say "somewhat difficult" or "reasonably difficult"
> 
> But as far as the overall argument is concerned, the important point
is
> that
> 
> "_very_ difficult" != "impossible" and "impossible" is tough to show.
> 
> The thing that limits the difficulty is that when things happen that
> are incompatible
> with your current assumption (as far as server identity) about a pair
> of IP addresses,
> you don't have any more uncertainty to deal with.  The algorithms are
> different if
> you start with the assumption of server non-identity and allow proof
of
> identity or
> or start by assuming identity when clientid4's match and allow proof
of
> non-identity,
> but the problem seems workable to me.  But I didn't say that in he
> spec. I said you may
> have problems with this and should make your own implementation
> choice.
> 
> > but only the server
> > knows that the client is accessing the same file through two
> different
> > IP addresses.
> 
> The client doesn't know that but when he gets the same clientid4 from
> two
> different IP addresses he has got to know that it is likely that they
> are
> the same.  Certainly it is possible even if it is not a sure thing.
> 
> > it will see changes
> > that appear to be in violation of the write delegation.
> 
> What it will see is a change that is in conflict with
> 
>  "A has a rd delegation" & "A and B are different servers"
> 
> Which propositions we will call X and Y respectively.
> 
> If something happens that is incompatible with X & Y then if X is
true,
> Then Y is false and so now the client knows hat A and B are the same.
> 
> Of course the delegation D might have been revoked so you use it on a
> Read to A.  If that is OK then you know Y is false, which you can
> Verify by doing the read using D over address B.
> 
> You can't do this with a vanilla client implementation but some people
> may
> choose Cookie Dough ice cream, and the spec allows that.
> 
> Another way to do this is to reduce the period of uncertainty by
> resolving server
> identity questions as soon as possible.  Once you get the first
stateid
> from a
> server, use it on all IP addresses sharing the same clientid4.  Those
> that recognize it
> Are connected to the same server.
> 
> I think it is becoming clear why v4.1 was defined to make this easier,
> but it
> still isn't impossible and if you find it onerous, you are free to
> choose the
> non-uniform model.
> 
> Summing up where we are:
> 
>    do you still object to MAY here?
> 
>    If so, I think we need a solid reason to justify "SHOULD" (for the
> non-uniform model).
>    Anyone want to take a shot at this?
> 
>    Or I it just that we differ on the tone of the discussion of the
> uniform model?
> 
>    If so, tell me what you'd like to see and I think we can work
toward
> something
>    reasonable.

I think that in practice the client does need to know a priori whether
or not it is talking to the same server through 2 different IP addresses
in order to make the "uniform model" a viable solution.

Once I've decided to mount what I think are 2 different filesystems,
then attempting to merge them at a later stage is impractical due to
general VFS design limitations (at least in Linux that is the case).
Once I get round to actually opening files, I will already have 2
separate trees of cached files/directories that are attached to 2
different super blocks, and so I'd have to do all sorts of distasteful
surgery in order to transparently merge those into a single tree, and
then migrate locks, migrate file descriptors, etc.
If I were rather to assume that the filesystems are the same, and then
try to fork them at a later stage when I find out that they are in fact
different, then I worry that could under some (admittedly rare)
circumstances lead to data corruption: e.g. if 2 filehandles that look
the same turn out to belong to 2 different servers. Again, the forking
process is not trivial to implement in practice.

IOW: I would find it hard to justify implementing all this extra
machinery in order to get a migration solution for NFSv4.0, particularly
given the fact that NFSv4.1 already offers me a way to avoid the entire
problem.

How about something like the following suggestion?

1) Servers that support trunking SHOULD allow a client to determine all
ports on which the server is listening using a COMPOUND of the form
PUTROOTFH; GETATTR fs_locations.

2) A client that want to use the "uniform model" MUST query fs_locations
on the server root filehandle prior to calling
SETCLIENTID/SETCLIENTID_CONFIRM. If the server does not support
fs_locations on the rootfh, then the client SHOULD use a unique long
form client id and MAY assume that the server does not support
migration(?), OR that a migration event must follow the non-uniform
model(?). Otherwise, the client may use RENEW requests on all the IP
addresses in order to discover on which ones a given lease applies.

Cheers
  Trond

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Trond Myklebust
> Sent: Tuesday, August 30, 2011 2:42 PM
> To: Noveck, David
> Cc: [email-masked]
> Subject: Re: [nfsv4] Individual i-d on v4.0 migration issues
> 
> On Tue, 2011-08-30 at 13:35 -0400, [email-masked] wrote:
> > Just wanted everybody to be aware of the attached.
> >
> >
> >
> > It is in the informational category, although its premise, that
> things need to be fixed in subsequent standards-track documents does
> have normative implications.
> >
> > I think it is now time to discuss the document and its implications
> within the working group.
> >
> 
> In Section 3.2.1 you debunk the text that is currently in RFC3530/bis
> w.r.t. whether or not a SETCLIENTID will cause a reboot if you reuse
> the
> same long form clientid to the same server on 2 different IP
addresses.
> While I agree with your argument, I don't think that it is sufficient,
> and that your 'uniform model' cannot work.
> 
> The main problem with a client that is unaware of the fact that it is
> talking to the same server over 2 different IP addresses is the fact
> that if it mounts the same filesystem over both addresses, it may end
> up
> performing conflicting stateful operations.
> 
> IOW: If my client OPENs a file on one IP address using a particular
> open
> owner string, then OPENs the same file on the other IP address using
> the
> same open owner string, then that client is going to find tracking the
> seqids to be _very_ difficult, since it thinks it is working with 2
> different seqid streams because it thinks it is dealing with 2
> different
> servers.
> 
> Even if the client uses different open owner strings in the above
> example, then it can run into trouble: for instance, if the server
> decides to give out a write delegation. A write delegation is supposed
> to guarantee that the client can cache aggressively, but only the
> server
> knows that the client is accessing the same file through two different
> IP addresses. As far as the client is concerned, it will see changes
> that appear to be in violation of the write delegation.
> Also, what happens in the case of a delegation recall? How does the
> client know that the recall applies to _both_ filesystems (assuming
> that
> the server handed out the same delegation to both)?
> 
> Cheers
>   Trond
> --
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> [email-masked]
> www.netapp.com
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 1 Sep 2011 14:22:48 -0700 (PDT)

> addresses in order to discover on which ones a given lease applies.

"PUTROOTFH; GETATTR fs_locations" can't be anything magical -
that is the proper way for a current NFSv4 client to inquire
about alternate replicas of this filesystem, in the complete
absence of migration.  We have implemented support for this.

Rob T

---

From: <[email-masked]>
Date: Thu, 1 Sep 2011 17:48:12 -0400

Nothing in the protocol is magical.  It's all just engineering.

Some things can appear magical, though, and while I don't think
Trond claimed anything special and magical, I did note his suggestion
and say to myself, "That is awfully neat".

It is true that this is a standard (although optional) way to find
out about alternate replicas, but I thought it was neat when applied
to the root (pseudo)-fs.  In that case, I think it is fair to
conclude that you can't have two different servers which have 
a replicated pseudo-fs, and that it is fair to conclude that you
have (non-magically) found out that there are multiple IP addresses
connected to the same server. 

Part of what made this cleverness appear magical is the set-up.
I have seen numerous assertions that (unlike in v4.1) it is impossible
to determine when two IP addresses are connected to the same server.

Maybe this is all part of the setup for Trond's magic trick :-) 

-----Original Message-----
From: Robert Thurlow [mailto:[email-masked]] 
Sent: Thursday, September 01, 2011 5:23 PM
To: Myklebust, Trond
Cc: Noveck, David; [email-masked]
Subject: Re: [nfsv4] Individual i-d on v4.0 migration issues

Rob T

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 01 Sep 2011 17:55:44 -0400

Hi Dave,

My turn to clarify :-)

      * Ultimately, I'd actually prefer to use the uniform model if it
        can be made a practical proposition. It seems to me that it
        solves several of the problems in a more elegant way (as opposed
        to adding yet more 'magic' combinations of operations). In
        addition, it is close enough to what we want to do for v4.1. My
        criticism is intended to highlight what I see as problems in the
        hope that we can work them out (and if you agree with the rootfh
        fs_locations thing, then I think we have addressed my main
        concern). 
      * I fully agree that lease merging is the underlying problem, and
        I don't see too many other ways to have a robust solution for
        migration without it. The only other solution I can think of is
        to have the v4 client set up one lease per filesystem, which is
        costly, and is really optimising for the (hopefully extremely
        infrequent) migration case. 
      * I still do have a concern with the existence of the magic
        combinations of ops (which are endemic to both the uniform and
        non-unform models thanks to the requirements implicit in
        LEASE_MOVED). If there were no clients out there that supported
        migration, then all would be fine - we could do the same thing
        that we did for referrals. My worry is that AFAICR, the AIX
        client claimed to have solved the issue of migration many years
        ago; are we now breaking their client assumptions (and will we
        incur the wrath of the IBM lawyers ;-))?

Cheers
  Trond

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 01 Sep 2011 17:56:18 -0400

> absence of migration.  We have implemented support for this.

That is 100% intentional. I'm just trying to formulate a way of
implementing the uniform model that can deal with multi-homed servers
(which I still see as a problem) _without_ introducing any new magic or
unexpected uses of the protocol.

The only new thing, is the implication that multi-homed servers that
want to support migration MUST do the above so that clients that want to
use the uniform model can detect when they are trunking.

Cheers
  Trond

NetApp

---

From: <[email-masked]>
Date: Thu, 1 Sep 2011 18:40:58 -0400

When I decided that we needed to support both models I anticipated debate with
arguments for the non-uniform-only position and also with those for the
uniform-only position.

What I didn't anticipate that was that both sorts of debate would be with the
same person.

I agree that the uniform approach is more elegant and also think that debate on
what is practical has no obvious stopping point.

I had hoped that the advocates of uniform and non-uniform would each have a client
to test, but I can't ask Trond to produce two clients.  One would be nice, though.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Trond Myklebust
Sent: Thursday, September 01, 2011 5:56 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] Individual i-d on v4.0 migration issues

Hi Dave,

My turn to clarify :-)

Cheers
  Trond

NetApp

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 01 Sep 2011 16:53:11 -0600

> That is 100% intentional.

I am sleep- and/or coffee-deprived today - it's the pseudo-root,
not (usually) the root of a real file system.  That helps, and
I'm warming to it.  We'd need to think about what to do if the
server's root FS is shared, though (I know, ick anyway).

Rob T

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 1 Sep 2011 22:56:45 +0000

Is there any need to differentiate multi-homed, single server
from multi-interface, multi-server (clustered) in this scenario?
Does anyone care at this point?

Spencer

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Robert Thurlow
> Sent: Thursday, September 01, 2011 3:53 PM
> To: Trond Myklebust
> Cc: [email-masked]
> I am sleep- and/or coffee-deprived today - it's the pseudo-root, not
> (usually) the root of a real file system.  That helps, and I'm warming to
> it.  We'd need to think about what to do if the server's root FS is
> shared, though (I know, ick anyway).
> 
> Rob T

---

From: Thomas Haynes <[email-masked]>
Date: Thu, 1 Sep 2011 15:56:58 -0700

> server's root FS is shared, though (I know, ick anyway).

This shouldn't be a problem, right?

PUTPUBFH; GETATTR fs_locations

:->

Although we won't get an answer from the Ann Arbor folks for a while
as they all did 

putPUBfh; OPEN "beer"; READ

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 01 Sep 2011 20:54:05 -0400

> Does anyone care at this point?

No. I think we have a test that is adequate to distinguish the two. If
done correctly, it is highly unlikely that a server would pass both
tests: fs_locations on the rootfs and RENEW.

Even if it always passes the first test, it should be trivial to ensure
that the multi-server passes/fails the second test depending on whether
it shares leases and other state between the front-end NFS servers or
not.

Cheers
  Trond

NetApp

---

From: Chuck Lever <[email-masked]>
Date: Fri, 2 Sep 2011 10:25:04 -0400

> to test, but I can't ask Trond to produce two clients.  One would be nice, though.

To be clear, Dave, I'm providing the Linux client-side migration implementation.  Very soon I will be working up a non-uniform prototype (based on the migration implementation I already have) to test against Bill's server prototype.

I'm also interested in trying a uniform client ID implementation, but the uniform client ID model looked like it had some issues to be worked out first, so we started with non-uniform.

I'm hopeful we will have the opportunity to visit both models.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: <[email-masked]>
Date: Fri, 2 Sep 2011 16:36:46 -0400

How would you tell the difference?  Is there any difference for the
sufficiently clustered multi-server?

If there is no behavioral test that shows a difference, then what the
server hardware looks like should not matter to the protocol.  In other
words, we should let "ontology recapitulate phenomenology" :-)

So the basic test for replication is that they show the same directories
Which have the same fileid's and fh's.  Both should show that.

Trond is assuming other things that clustered implementation should show
In the specific case of the pseudo-fs.

    The same nfs4_client_id should result in he same clientid4.

    When the same clientid4's are present at multiple ip addresses that a stateid
    received from one should be recognized on the other.

    Similarly, if I use the same id with a different verifier, I can invalidate
    a previously obtained clientid4 from another interface (good as it is another    
    test that can be done before open). 

I think that the spec should be clear whether or not the multiple addresses are
being certified by the server as meeting these tests or not, and if not, the client 
Can and should check himself. 

I think it makes sense to say that for this case (the pseudo-fs) the multiple
addresses should meet the same requirements that v4.1 applies for returning matching
server-major values.

Note that in v4.1, addresses with the same server-major and different server-minor 
addresses can share state but they are not sufficiently clustered that they can share
the same session.

V4.0 doesn't have sessions but I think we can assume/state that the server(s) may treat
the addresses as having distinct reply caches, i.e. that the client may not depend on
their being a shared reply cache.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Spencer Shepler
Sent: Thursday, September 01, 2011 6:57 PM
To: Robert Thurlow; Trond Myklebust
Cc: [email-masked]
Subject: Re: [nfsv4] Individual i-d on v4.0 migration issues

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 02 Sep 2011 17:09:47 -0400

> addresses should meet the same requirements that v4.1 applies for returning matching

What if PUTROOTFH points to a real filesystem (as might be the case for
a pure NFSv4 server)? I think it makes sense to apply the same rules and
assumptions, but I thought I'd raise the point.

In any case, I think the client should try a RENEW before issuing the
SETCLIENTID if the PUTROOTFH+GETATTR fs_locations turns up an IP address
that is recognisable as one on which the client knows that it already
holds a lease (and therefore has a short form clientid).

> their being a shared reply cache.

That would mean that the open_owners (and hence stateids) cannot be
shared, since the open_owner seqids form the basis for the reply cache
in NFSv4.0 (to allow me to replay OPEN, LOCK,... commands that got
lost). Wouldn't that be different for the multi-interface single server
case?

...or are you only referring to the NFSv2/v3 style of XID-based DRC?

Cheers
  Trond

NetApp

---

From: <[email-masked]>
Date: Sat, 3 Sep 2011 15:07:28 -0400

> ...or are you only referring to the NFSv2/v3 style of XID-based DRC? 

That's what I was referring to.

As far as clustering requirements, there is a big difference between a co-ordination/clustering requirement that you have to apply on every request and one that you only apply on state-related requests.  That's the reason the server-major/server-minor distinction was drawn that way for v4.1 and I'd argue that it is an appropriate requirement to put on clustered servers that need to seem as much like a single server as can reasonably be expected.

In other words, that is my definition of "reasonably" in the above.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Trond Myklebust
Sent: Friday, September 02, 2011 5:10 PM
To: Noveck, David
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] Individual i-d on v4.0 migration issues

Cheers
  Trond

NetApp