Subject: [nfsv4] NFSv4.0 migration issues

From: Chuck Lever <[email-masked]>
Date: Mon, 8 Aug 2011 14:43:41 -0400

As some may know, the Oracle Solaris team and the Linux NFS community have implemented NFSv4.0 migration.  Linux currently has a client implementation, Solaris has both client and server.  During implementation, we've encountered some challenges with sections of 3530 which deal with migration.

We reported on some of these issues at the interim NFSv4 WG meeting in February.  This is a follow-up with additional concerns, motivated by discussions at the recent IETF 81 meeting in Quebec City.

Given: An NFSv4.0 client identifies itself to servers using an opaque string called an nfs4_clientid.  An nfs4_clientid is typically formed with, among other things, the presentation IP addresses of the client and server that are establishing a lease relationship.  Below, suppose the client IP address is C, source server IP address is Ss, and the destination server IP address is Sd.

Additionally, assume that a migration involving transparent state migration (TSM) moves data, FSID, leases, the short client ID, and the nfs4_clientid from the source server to the destination server.  When establishing a lease relationship with the destination server during migration recovery, the client must perform a SETCLIENTID operation (which takes an nfs4_clientid as an argument) to update the destination with the client's callback information.

Problem: What nfs4_clientid does a client use for this SETCLIENTID operation?

If the client uses [C, Ss] (the nfs4_clientid migrated from the source server) the destination server can easily match the nfs4_clientid sent by the client to a previously migrated lease.  However, if the client reboots, it will not remember that migration has occurred, and will thus re-identify itself to the destination server by its natural nfs4_clientid [C, Sd].  The destination server will not be able to match this new nfs4_clientid to any existing leases.  The migrated lease will have to expire before the client can access migrated files.

If the client uses [C, Sd] (the natural nfs4_clientid the client might use with the destination server), the destination server has no way to match this nfs4_clientid to a migrated lease, unless it parses the opaque string to discover the client's IP address, which is frowned upon.  Thus transparent state migration is not possible without breaching spec-compliance.  However, the client can reboot, and it can reestablish leases it has on the destination server.

Problem: the section of 3530 that discusses the SETCLIENTID_CONFIRM operation appears to preclude the ability of a client to perform a callback update while preserving state migrated via TSM.  RFC 3530 section 14.2.34 says:

"Otherwise, the server records a confirmed { *, x, c, k, s }.  If there is also a confirmed { *, x, d, *, t }, the server MUST remove the client x's relevant leased client state, and overwrite the callback state with k.  The confirmed record { *, x, d, *, t } is removed.

Server returns NFS4_OK."

This paragraph is designed to prevent a SETCLIENTID_CONFIRM replay.  However, it forces a spec-compliant destination server to drop migrated state when a client performs SETCLIENTID_CONFIRM after a callback update. Thus the client never has access to migrated state, and TSM is impossible.

The language in 3530bis is the same, I believe.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Rick Macklem <[email-masked]>
Date: Mon, 8 Aug 2011 18:42:25 -0400 (EDT)

Chuck Lever wrote:
> As some may know, the Oracle Solaris team and the Linux NFS community
> have implemented NFSv4.0 migration. Linux currently has a client
> implementation, Solaris has both client and server. During
> implementation, we've encountered some challenges with sections of
> 3530 which deal with migration.
> 
> We reported on some of these issues at the interim NFSv4 WG meeting in
> February. This is a follow-up with additional concerns, motivated by
> discussions at the recent IETF 81 meeting in Quebec City.
> 
> 
> Given: An NFSv4.0 client identifies itself to servers using an opaque
> string called an nfs4_clientid. An nfs4_clientid is typically formed
> with, among other things, the presentation IP addresses of the client
> and server that are establishing a lease relationship. Below, suppose
> the client IP address is C, source server IP address is Ss, and the
> destination server IP address is Sd.
> 
My understanding is that the nfs_client_id4.id (what I assume you mean
when you say nfs4_clientid) should be a unique identifier for the client.
I don't see any reason why the server's IP address should be in it.

I also believe that having the client's IP address in it is a poor choice
as well. After all, an IP address assigned dynamically via DHCP can change.
Or, what about a client that sits behind a NAT gateway, so it always gets
192.168.n.n as an address. That's hardly unique, is it?

I think using something like a hardware MAC address for a net interface in
the client is a better choice. On FreeBSD, the UUID for the machine is used.
(The algorithm for generating the UUID looks involved, but appears to be
 described in some open group document and looks like it uses MAC address(es)
 among other things.)

Seems to me, the problem is a poor choice of nfs_client_id4.id to me?

> Additionally, assume that a migration involving transparent state
> migration (TSM) moves data, FSID, leases, the short client ID, and the
> nfs4_clientid from the source server to the destination server. When
> establishing a lease relationship with the destination server during
> migration recovery, the client must perform a SETCLIENTID operation
> (which takes an nfs4_clientid as an argument) to update the
> destination with the client's callback information.
> 
It should always use the same nfs_client_id4.id (the nfs_client_id4.verifier
needs to change when the client does a reboot or similar) as far as I
understand the protocol. Just choose one that doesn't have IP address(es)
in it.

> If the client uses [C, Ss] (the nfs4_clientid migrated from the source
> server) the destination server can easily match the nfs4_clientid sent
> by the client to a previously migrated lease. However, if the client
> reboots, it will not remember that migration has occurred, and will
> thus re-identify itself to the destination server by its natural
> nfs4_clientid [C, Sd]. The destination server will not be able to
> match this new nfs4_clientid to any existing leases. The migrated
> lease will have to expire before the client can access migrated files.
> 
> If the client uses [C, Sd] (the natural nfs4_clientid the client might
> use with the destination server), the destination server has no way to
> match this nfs4_clientid to a migrated lease, unless it parses the
> opaque string to discover the client's IP address, which is frowned
> upon. Thus transparent state migration is not possible without
> breaching spec-compliance. However, the client can reboot, and it can
> reestablish leases it has on the destination server.
> 
> 
> Problem: the section of 3530 that discusses the SETCLIENTID_CONFIRM
> operation appears to preclude the ability of a client to perform a
> callback update while preserving state migrated via TSM. RFC 3530
> section 14.2.34 says:
> 
> "Otherwise, the server records a confirmed { *, x, c, k, s }. If there
> is also a confirmed { *, x, d, *, t }, the server MUST remove the
> client x's relevant leased client state, and overwrite the callback
> state with k. The confirmed record { *, x, d, *, t } is removed.
> 
> Server returns NFS4_OK."
> 
> This paragraph is designed to prevent a SETCLIENTID_CONFIRM replay.
> However, it forces a spec-compliant destination server to drop
> migrated state when a client performs SETCLIENTID_CONFIRM after a
> callback update. Thus the client never has access to migrated state,
> and TSM is impossible.
> 
Nope, it just assumes the nfs_client_id4.id won't change, which seems
reasonable to me.

rick

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 09 Aug 2011 04:38:31 -0400

> The language in 3530bis is the same, I believe.

If you want to keep 2 _different_ leases, they need to be identified by
_different_ clientids. Anything else leads to insanity.

The whole problem above arises from the fact that you are treating these
migrations as creating new leases on the target. Why is that necessary?

Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Rick Macklem <[email-masked]>
Date: Tue, 9 Aug 2011 08:33:24 -0400 (EDT)

Trond Myklebust wrote:
> > As some may know, the Oracle Solaris team and the Linux NFS
> > community have implemented NFSv4.0 migration. Linux currently has a
> > client implementation, Solaris has both client and server. During
> > We reported on some of these issues at the interim NFSv4 WG meeting
> > in February. This is a follow-up with additional concerns, motivated
> > by discussions at the recent IETF 81 meeting in Quebec City.
> >
> >
> > Given: An NFSv4.0 client identifies itself to servers using an
> > opaque string called an nfs4_clientid. An nfs4_clientid is typically
> > formed with, among other things, the presentation IP addresses of
> > the client and server that are establishing a lease relationship.
> > Below, suppose the client IP address is C, source server IP address
> > is Ss, and the destination server IP address is Sd.
> >
> > migration (TSM) moves data, FSID, leases, the short client ID, and
> > the nfs4_clientid from the source server to the destination server.
> > When establishing a lease relationship with the destination server
> > during migration recovery, the client must perform a SETCLIENTID
> > operation (which takes an nfs4_clientid as an argument) to update
> > the destination with the client's callback information.
> >
> > If the client uses [C, Ss] (the nfs4_clientid migrated from the
> > source server) the destination server can easily match the
> > nfs4_clientid sent by the client to a previously migrated lease.
> > However, if the client reboots, it will not remember that migration
> > has occurred, and will thus re-identify itself to the destination
> > server by its natural nfs4_clientid [C, Sd]. The destination server
> > will not be able to match this new nfs4_clientid to any existing
> > leases. The migrated lease will have to expire before the client can
> > access migrated files.
> >
> > If the client uses [C, Sd] (the natural nfs4_clientid the client
> > might use with the destination server), the destination server has
> > no way to match this nfs4_clientid to a migrated lease, unless it
> > parses the opaque string to discover the client's IP address, which
> > is frowned upon. Thus transparent state migration is not possible
> > without breaching spec-compliance. However, the client can reboot,
> > and it can reestablish leases it has on the destination server.
> >
> >
> > "Otherwise, the server records a confirmed { *, x, c, k, s }. If
> > there is also a confirmed { *, x, d, *, t }, the server MUST remove
> > the client x's relevant leased client state, and overwrite the
> > callback state with k. The confirmed record { *, x, d, *, t } is
> > removed.
> >
> > Server returns NFS4_OK."
> >
If the migration is to the somewhere else on the same server then, yes,
the nfs_client_id4.id needs to be different. I had assumed the migration
was to a different server. (I haven't looked at the migration specs, so
if TSM is restricted to "same server", then apologies for the noise.)

> migrations as creating new leases on the target. Why is that

If the migration is to a different server, then I don't see why the
same nfs_client_id4.id couldn't be used to change the callback address.
(I had assumed a different server which needed to talk to the client
 via a different callback address, due to network topology. Again, if
 TSM is restricted to "same server", what I've been typing is bogus noise
 that should be ignored.)

rick

---

From: Chuck Lever <[email-masked]>
Date: Tue, 9 Aug 2011 10:48:46 -0400

> _different_ clientids. Anything else leads to insanity.

On the other hand, if the client reboots but doesn't remember its migrated client IDs, it will not be able to reconnect to the migrated lease on the destination server.

> migrations as creating new leases on the target. Why is that necessary?

Do you mean that if there is already an existing lease relationship between the client and the destination server, the destination server should merge the migrated lease with an existing lease?  If so, we believe it is challenging for the destination server to identify precisely which two leases to merge in all cases.

The client would also have to know in advance which existing lease to look for on the destination server.  If there is only zero or one lease relationship already, that would be straightforward.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 9 Aug 2011 10:56:37 -0400

> I don't see any reason why the server's IP address should be in it.

Pages 66 and 67 of RFC 3530 describe how a client should construct the nfs_client_id4 string.  Bullet point 3 has this language:

"The string should be different for each server network address that the client accesses, rather than common to all server network addresses. The reason is that it may not be possible for the client to tell if the same server is listening on multiple network addresses. If the client issues SETCLIENTID with the same id string to each network address of such a server, the server will think it is the same client, and each successive SETCLIENTID will cause the server to begin the process of removing the client’s previous leased state."

Whether the client uses the server's IP address or something else that is unique to that server to construct nfs_client_id4, we still have the same problem.  Which nfs_client_id4 string should the client use when contacting the destination server?

> described in some open group document and looks like it uses MAC address(es)

Using a MAC address instead of the client's IP address seems reasonable, but that doesn't address the above problem.

> Seems to me, the problem is a poor choice of nfs_client_id4.id to me?

The client is left with unfortunate alternatives.

---

From: Bill Baker <[email-masked]>
Date: Tue, 09 Aug 2011 10:27:24 -0500

> migrations as creating new leases on the target. Why is that necessary?

It is necessary because the receiving server cannot figure out how to
merge the state with any existing lease it might already have from that
client (without hints, more on that later).

The TSM source server has a state tree rooted by the long form client
ID (which has an embedded server IP).  When this state tree comes over,
the server cannot figure out how to associated this state tree with an
existing state tree from the same client because the client long form
is different.

In our prototype implementation, we solved this problem
by having 2 (or more leases).  Yes, this leads to other problems.

As part of our ongoing investigation of this problem, we've discussed
a merged lease solution too, and it is NOT without problems.  Based
on some off-list discussion, we believe Dave Noveck has come up with
a solution, whereby, with client help and some creative application of
the protocol, the merged lease solution is workable.  We have not
attempted to implement this solution.

Chuck's original email on this topic was meant to set the stage for
the introduction of Dave's proposed solution.  Look for that shortly.

-- 
Bill Baker, Oracle Solaris NFS development

---

From: Chuck Lever <[email-masked]>
Date: Tue, 9 Aug 2011 11:43:21 -0400

> Pages 66 and 67 of RFC 3530 describe how a client should construct the nfs_client_id4 string.  Bullet point 3 has this language:

Side note: This is on page 108 and 109 of 3530bis-12.  The text looks the same as RFC 3530.

> "The string should be different for each server network address that the client accesses, rather than common to all server network addresses. The reason is that it may not be possible for the clien…

The text goes on to recommend the use of both the client and server network address when constructing an nfs_client_id4 string.  Essentially, either we should fix this recommendation, or find a way to make NFSv4.0 transparent state migration work with the server IP address in this string.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Rick Macklem <[email-masked]>
Date: Tue, 9 Aug 2011 20:49:56 -0400 (EDT)

Chuck Lever wrote:
> >>> community
> >>> opaque
> > Pages 66 and 67 of RFC 3530 describe how a client should construct
> > the nfs_client_id4 string. Bullet point 3 has this language:
> 
> > "The string should be different for each server network address that
> > the client accesses, rather than common to all server network
> > addresses. The reason is that it may not be possible for the client
> > to tell if the same server is listening on multiple network
> > addresses. If the client issues SETCLIENTID with the same id string
> > to each network address of such a server, the server will think it
> > is the same client, and each successive SETCLIENTID will cause the
> > server to begin the process of removing the client’s previous leased
> > state."
> 
Just fyi, the FreeBSD client uses a different nfs_client_id4.id string for
each mount point, so it does avoid duplication for cases that "might be the
same server". (ie. It maintains separate clientid/stateid spaces for each mount
point, even if it thinks that it is the same server, to avoid the "might be the
same server" issue.) However, it does not use IP address(es) in the id.

> The text goes on to recommend the use of both the client and server
> network address when constructing an nfs_client_id4 string.
> Essentially, either we should fix this recommendation, or find a way
> to make NFSv4.0 transparent state migration work with the server IP
> address in this string.
> 
Personally, I don't think the example of a good nfs_client_id4.id is
all that good, but it is just an example... (It's picking nits, but
I'd consider an example as weaker than a recommendation.)

In particular, it mentions that the IP address of the client may change
just before it suggests putting it in the string, without justification
of why that's a good thing to do.

I haven't looked at the migration stuff, so I should probably just
shut up now, but I can see there is a problem here, in that it seems
the client somehow needs to decide if it is a "different server"
(I'd define "different server" as one which maintains a separate
clientid/stateid space.) that has been migrated to.
Same server -> use extant clientid, etc
Different server -> you can use same nfs_client_id4.id string, since
   the bullet point w.r.t. "might be same server" doesn't apply.

I don't know if the migration document discusses this or not?
(I think a "different server" would reply NFS4ERR_BAD_CLIENTID/
 NFS4ERR_BAD_STATEID when a clientid/stateid issued before the
 referral is followed, is used. However, I'm not sure if that
 would be sufficient?)

rick, who will now shut up until he reads the migration stuff someday.
(Apologies for the noise.)

---

From: <[email-masked]>
Date: Tue, 9 Aug 2011 21:56:17 -0400

That's not the way I read the current spec.

   How do you read the current spec?

   With great difficulty :-)

Still I think the following is what is intended.

I think the SETCLIENTID is covered by:

   o  The server checks if it has recorded a confirmed record for { v,
      x, c, l, s }, where l may or may not equal k.  If so, and since
      the id verifier v of the request matches that which is confirmed
      and recorded, the server treats this as a probable callback
      information update and records an unconfirmed { v, x, c, k, t }

Haynes & Noveck         Expires October 10, 2011              [Page 293]

Internet-Draft                    NFSv4                       April 2011

      and leaves the confirmed { v, x, c, l, s } in place, such that t
      != s.  It does not matter if k equals l or not.  Any pre-existing
      unconfirmed { v, x, c, *, * } is removed.

      The server returns { c, t }.  It is indeed returning the old
      clientid4 value c, because the client apparently only wants to
      update callback value k to value l.  It's possible this request is
      one from the Byzantine router that has stale callback information,
      but this is not a problem.  The callback information update is
      only confirmed if followed up by a SETCLIENTID_CONFIRM { c, t }.

      The server awaits confirmation of k via SETCLIENTID_CONFIRM { c, t
      }.

      The server does NOT remove client (lock/share/delegation) state
      for x.

And the confirm is covered by:

   o  The server has recorded an unconfirmed { v, x, c, k, s } record
      and a confirmed { v, x, c, l, t } record, such that s != t.  If
      the principals of the records do not match that of the
      SETCLIENTID_CONFIRM, the server returns NFS4ERR_CLID_INUSE, and no
      relevant leased client state is removed and no recorded callback
      and callback_ident information for client { x } is changed.
      Otherwise, the confirmed { v, x, c, l, t } record is removed and
      the unconfirmed { v, x, c, k, s } is marked as confirmed, thereby
      modifying recorded and confirmed callback and callback_ident
      information for client { x }.

      The server does not remove any relevant leased client state.

      The server returns NFS4_OK.

If someone thinks I'm wrong, explain to me why it doesn't work as I think it does.

If someone is still unsure, but thinks that it should work as I think it does, please suggest
clarifications that would assure people that no state is removed.  I think everyone
agrees that state shouldn't be removed (i.e. that removing it would be a bad thing). 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Trond Myklebust
Sent: Tuesday, August 09, 2011 4:39 AM
To: Chuck Lever
Cc: [email-masked]
Subject: Re: [nfsv4] NFSv4.0 migration issues

Trond

NetApp

---

From: <[email-masked]>
Date: Wed, 10 Aug 2011 09:42:31 -0400

I'm not going to continue in nostalgie-de-la-deja-vu style
(see what Quebec City did to me :-), but I agree with you that there  
is something very wrong in what is there now.
 
With regards to your specific remarks, I don't see a way of distinguishing same-server vs. different-server in v4.0.  it is in v4.1 but unless you have some really great idea, I don't see this happening in v4.0.

With regard to the bullet point applying or not applying in the different server case, I think the text is right that there is no way to know.  What I don't see is the negative consequences being assumed in the same-server case.  See the following:

   In point of fact, a "SETCLIENTID with the same id string" to multiple
   network addresses will be treated as all from the same client but it
   will not "cause the server to begin the process of removing the
   client's previous leased state", unless the server believes it is a
   newer instance of the same client, i.e. if the id is the same and has
   a different verifier.  If the client does not reboot, the verifier
   should not change and if it does, the verifier will change, and the
   server should "begin the process of removing the client's previous
   leased state".

   The situation of multiple SETCLIENTID requests received by a server
   on multiple network addresses, is exactly the same, from the protocol
   design point of view as that of multiple (i.e. duplicate) SETCLIENTID
   requests received by the server on a single network address and the
   same protocol mechanisms that prevent erroneous state deletion in the
   latter case prevent it in the former case.  There is thus no reason  
   for special handling of the multiple-network-appaerance case.

At least, that is the way I see it.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rick Macklem
Sent: Tuesday, August 09, 2011 8:50 PM
To: Chuck Lever
Cc: [email-masked]
Subject: Re: [nfsv4] NFSv4.0 migration issues

Chuck Lever wrote:
> > server to begin the process of removing the client's previous leased
> > state."
>

---

From: <[email-masked]>
Date: Wed, 10 Aug 2011 10:26:34 -0400

> Problem: What nfs4_clientid does a client use for this SETCLIENTID operation?

It uses the nfs_client_id4 that is associated (on the client) with he clientid4 that is migrated.

I never grasped why this is required, but that's beside the point.  The client has the information to do this if it should be done.

If the client has rebooted since then (and has no record of the migrated clientid4), then he shouldn't have any state and
should not try re-establishing a connection to state on the server that has no correlate on the client side.  Something has gone wrong and the focus should be on getting that state cleared.  He should do a SETCLIENTID, not to update callback information but to clear state.

So the problem here is that the nfs_client_id4 structure makes clearing state appropriately impossible and not that of updating callback information.  When you need to update callback information you have the ability to do it.  
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Chuck Lever
Sent: Monday, August 08, 2011 2:44 PM
To: [email-masked]
Subject: [nfsv4] NFSv4.0 migration issues

Server returns NFS4_OK."

---

From: Rick Macklem <[email-masked]>
Date: Wed, 10 Aug 2011 19:15:33 -0400 (EDT)

David Noveck wrote:
> With regards to your specific remarks, I don't see a way of
> distinguishing same-server vs. different-server in v4.0. it is in v4.1
> but unless you have some really great idea, I don't see this happening
> in v4.0.
> 
> With regard to the bullet point applying or not applying in the
> different server case, I think the text is right that there is no way
> to know. What I don't see is the negative consequences being assumed
> in the same-server case. See the following:
> 
I don't know if this has anything to do with the original issue related
to migration, but...

In the FreeBSD client case, I choose to do a SETCLIENTID/SETCLIENTIDCONFIRM
at dismount, with a different nfs_client_id4.verifier, so that any state
(should only be open_owners) is released at the server for that mount point.
Think of this as analagous to DESTROY_CLIENTID in 4.1. Because of this, the
client needs to have different nfs_client_id4.id strings for each mount.

If I did not do this, I think I agree with you that doing a SETCLIENTID/
SETCLIENTIDCONFIRM multiple times with the same nfs_client_id4 ("v" and "x" same)
seems harmless. At most, it might change the callback path (if the second
server IP address is routed through a different net interface on the client) to
a different one that should work. (Maybe the original author was concerned that
the change in callback address might be detrimental, although I don't really see
why that would be the case.)

If a client machine "plays it safe" and does RENEW ops for each server IP address,
then all that happens if the IP addresses refer to the same server is a few extra
RENEW operations.

I took a quick look and didn't see any internet-draft on migration for 4.0. I did
take a quick look at RFC5661, but that didn't really help a lot w.r.t. what you
guys are trying to do in 4.0. As such, here's a possibly way of the mark plan:
- The FreeBSD client uses an nfs_client_id4.id string that consists of
  <unique-mount-point-id + unique-client-machine-id>
  It maintains a separate clientid (lease state I think you call it) for each
  mount point.

My plan for what to do when a referral was encountered is (not yet implemented):
- Create a sub-mount which inherits the parent mount's properties except what
  is changed by the referral info. Almost always a new FH and often a different
  server IP address.
  - Do a SETCLIENTID/SETCLIENTIDCONFIRM using the same nfs_client_id4 (same "v" and "x")
    as the parent mount, with maybe a different callback path. (As above, if this
    happens to be the same server, it would be harmless noise. If it is a new server,
    it would establish a new clientid for the subtree.)
    The ClientID might/might not be the same bits, but that doesn't matter.
  - The client machine would do RENEW Ops for all mounts/sub-mounts with different
    server IP addresses. (That way, all leases should get renewed. If the server is
    the same, they just get renewed more often than necessary.)

Now, would this work for what you guys are planning to do at the server?

If not relevent, just ignore this noise, rick

> > >>> February. This is a follow-up with additional concerns,
> > >>> motivated
> > >>> by

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 11 Aug 2011 05:41:27 +0000

<in response to the state migration thread>

The following is my attempt to summarize the discussion.
Comments and corrections welcome.

------

When receiving a callback, the client needs a method to determine
which server is making the call.  This is achieved through the use of
the callback_ident field in CB_COMPOUNDargs.

The reason the client needs this identification is that filehandles
are only guaranteed unique within a single server.  From the
callback_ident value, the client is then able to scope the filehandle
search correctly.

This implies the client must use a different callback_ident for each
unique server.

The callback_ident field is part of the SETCLIENTID call and is part
of the "callback" information.

In NFSv4.0, no method exists for the client to determine server
identity.  Thus, if the server has 2 or more IP addresses, the client
must assume each of those addresses is a unique server.  If the client
were to keep the value of nfs_client_id4 the same for each SETCLIENTID
to each of those address but vary the callback_ident value (as it
must), the server will properly identify the client as the same client
previously confirmed and will reset the callback information.  The
client is then left without the unique identification needed to
de-multiplex any potential callbacks.

To avoid the "reset" of the needed callback_ident, the client must
then present a unique nfs_client_id4 value for each server.

To allow for quick recovery (removal of pre-existing state at the
server), the client should use a value for nfs_client_id4 that
persists between instances of the client.  A somewhat conflicted
discussion exists in section 9.1.1 of rfc3530bis about choices of
values for nfs_client_id4.

For the client to take advantage of the transparent state migration,
the client must present the same nfs_client_id4 value (and all related
state values) to the new server.  Updating the callback information is
desirable.  For the client to have "fast" recovery (release of state)
upon restart, it must use the same nfs_client_id4 value used in the
previous instance of the client.

If the client uses a server-identity-artifact in the nfs_client_id4
and that artifact changes when a migration event occurs, the client
is: a) going to lose access to the migrated state, or b) going to lose
the "fast" recovery upon restart.

If the client is able to create an nfs_client_id4 value that meets the
various requirements of client uniqueness, can move with the migration
events, etc. we are golden.  That seems problematic in that no
suggestions have been made about how to effectively achieve this.

No method exists to "transfer" this state between nfs_client_id4
values in NFSv4.0.

----

At this point, it seems that section 9.1.1 needs a little work to
clean up the conflicting advice.  Less prescriptive and more
descriptive of the requirements about nfs_client_id4 values would be
helpful.

Barring suggestions about creation of an appropriate nfs_client_id4
values, it seems that for transparent state migration events, the
client will have to wait while the leased state times out and is
released.

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 11 Aug 2011 10:24:38 -0400

> agrees that state shouldn't be removed (i.e. that removing it would be a bad thing). 

I'm probably missing something (I'm being a bit distracted by other
events this week and am only keeping half an eye on this thread),
however the relevant passage in SETCLIENTID_CONFIRM that worries me is

"Otherwise, the server records a confirmed { *, x, c, k, s }. If there
is also a confirmed { *, x, d, *, t }, the server MUST remove the client
x's relevant leased client state, and overwrite the callback state with
k. The confirmed record { *, x, d, *, t } is removed."

which seems to imply that if you confirm a short hand clientid (with the
same long form clientid), then that automatically removes any
differently valued short hand clientids and all state that is associated
with those short hand clientids.

IOW: If you try to operate with multiple short hand clientids for the
same long form clientid (as has been proposed earlier by Chuck), then
you will sooner or later run into trouble.

Cheers,
  Trond

NetApp

---

From: Rick Macklem <[email-masked]>
Date: Thu, 11 Aug 2011 10:26:04 -0400 (EDT)

> then present a unique nfs_client_id4 value for each server.
Could this be resolved by "clarifying" the definition of the "server"
string within the fs_locations attribute to optionally have a "SAME/DIFFERENT"
indicator so that the client knows whether or not the new server is a
different one?
Something like:
    S:new-server.domain
    S:131.104.48.7
vs
    D:new-server.domain
    D:131.104.49.7

Or maybe the fs_locations.fs_root field could be re-defined to indicate
this? (I've never understood what it was used for anyhow, since there
isn't really a multi-component lookup?)

That way the client would know if a different callback_ident was needed
for this case.

Btw, thanks for the summary Spencer. I think I might understand what they
were talking about now:-)

rick

---

From: Chuck Lever <[email-masked]>
Date: Thu, 11 Aug 2011 11:16:50 -0400

> client will have to wait while the leased state times out and is

To be clear, this is only an issue after a client that has migrated some FSIDs has rebooted.  I would like to understand better the risks of waiting for an orphaned lease to time out.  That's potentially a big window.

---

From: <[email-masked]>
Date: Thu, 11 Aug 2011 12:49:37 -0400

Could you point out the "conflicting advice" in  section 9.1.1?  I see it conflicting with migration's needs but I don't see multiple conflicting recommendations.   

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Spencer Shepler
Sent: Thursday, August 11, 2011 1:41 AM
To: [email-masked]
Subject: Re: [nfsv4] NFSv4.0 migration issues

------

----

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 11 Aug 2011 17:21:37 +0000

As someone else pointed out, it suggests the client address not be
used because it may change and then suggests using the client 
network address.  Obviously, "network address" may refer to something
other than the DHCP IP address but even the MAC address may change
if in a virtualized environment that changes that MAC address between
reboots of the system.  Minor conflict but still present.

------

   o  The algorithm for generating the string should not assume that the
      client's network address won't change.  This includes changes
      between client incarnations and even changes while the client is
      stilling running in its current incarnation.  This means that if
      the client includes just the client's and server's network address
      in the id string, there is a real risk, after the client gives up
      the network address, that another client, using a similar
      algorithm for generating the id string, will generate a
      conflicting id string.

   Given the above considerations, an example of a well generated id
   string is one that includes:

   o  The server's network address.

   o  The client's network address.

> From: [email-masked] [mailto:[email-masked]]
> Sent: Thursday, August 11, 2011 9:50 AM
> To: Spencer Shepler; [email-masked]
> Subject: RE: [nfsv4] NFSv4.0 migration issues
> 
> Could you point out the "conflicting advice" in  section 9.1.1?  I see it
> conflicting with migration's needs but I don't see multiple conflicting
> recommendations.
> 
> 
> When receiving a callback, the client needs a method to determine which
> server is making the call.  This is achieved through the use of the
> callback_ident field in CB_COMPOUNDargs.
> 
> The reason the client needs this identification is that filehandles are
> only guaranteed unique within a single server.  From the callback_ident
> value, the client is then able to scope the filehandle search correctly.
> 
> The callback_ident field is part of the SETCLIENTID call and is part of
> the "callback" information.
> 
> In NFSv4.0, no method exists for the client to determine server identity.
> Thus, if the server has 2 or more IP addresses, the client must assume
> each of those addresses is a unique server.  If the client were to keep
> the value of nfs_client_id4 the same for each SETCLIENTID to each of those
> address but vary the callback_ident value (as it must), the server will
> properly identify the client as the same client previously confirmed and
> will reset the callback information.  The client is then left without the
> unique identification needed to de-multiplex any potential callbacks.
> 
> To avoid the "reset" of the needed callback_ident, the client must then
> present a unique nfs_client_id4 value for each server.
> 
> To allow for quick recovery (removal of pre-existing state at the server),
> the client should use a value for nfs_client_id4 that persists between
> instances of the client.  A somewhat conflicted discussion exists in
> section 9.1.1 of rfc3530bis about choices of values for nfs_client_id4.
> 
> For the client to take advantage of the transparent state migration, the
> client must present the same nfs_client_id4 value (and all related state
> values) to the new server.  Updating the callback information is
> desirable.  For the client to have "fast" recovery (release of state) upon
> restart, it must use the same nfs_client_id4 value used in the previous
> instance of the client.
> 
> If the client uses a server-identity-artifact in the nfs_client_id4 and
> that artifact changes when a migration event occurs, the client
> is: a) going to lose access to the migrated state, or b) going to lose the
> "fast" recovery upon restart.
> 
> events, etc. we are golden.  That seems problematic in that no suggestions
> have been made about how to effectively achieve this.
> 
> No method exists to "transfer" this state between nfs_client_id4 values in
> NFSv4.0.
> 
> ----
> 
> At this point, it seems that section 9.1.1 needs a little work to clean up
> the conflicting advice.  Less prescriptive and more descriptive of the
> requirements about nfs_client_id4 values would be helpful.
> 
> values, it seems that for transparent state migration events, the client
> will have to wait while the leased state times out and is released.
> 
> Spencer
>

---

From: <[email-masked]>
Date: Fri, 12 Aug 2011 09:35:28 -0400

OK.

If you look at this area currently, you find a straddling between prescriptive
and descriptive.   The word "should" along with "well-constructed" is an
indication that something is amiss.

Either something would be wrong and that you "SHOULD" or "MUST" do X to prevent
it, or there is an arguable benefit to doing X, in which "MAY" should tell the other 
party that people will be doing this, and that it can't rely on them not doing it.

So far, I have wrestled with this as far as bullet-point-3 in that description,
Where I'm leaning is toward simply saying "MAY" and will have some sort of approach
for what to say after that in the Monday copy.

Right now, my pre-draft spends a lot of time saying "utterly without merit" about 
the existing explanation for different nfs4_client_id4's that reflect server address,
but that is only because it is in RFC3530 and needs to be gotten rid of.

I don't think we want to get into a big discussion of Spencer's new explanation of this.
If everybody agrees that the spec will say "MAY" on this point, we can focus on
making sure that any descriptive follow-on is truly descriptive, and not 
disguised-prescriptive.  If you say "you should do this since you will have a mess
if you don't" you are really prescriptive and will have to face the question: "Why 
doesn't it say "SHOULD" or "MUST" then?". 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Spencer Shepler
Sent: Thursday, August 11, 2011 1:22 PM
To: [email-masked]
Subject: Re: [nfsv4] NFSv4.0 migration issues

------