Subject: SETCLIENTID issues (long)

From: Mike Eisler <[email-masked]>
Date: Fri, 19 Jul 2002 17:51:00 -0700

Having been responsible for original setclientid stuff,
I'll try to clean it up (with some help previously from
Dave N. and Spencer in the background this week).

Here are some observations/opinions (which Dave and Spencer
has contributed to):

- SETCLIENTID has a client incarnation value, known as
  the field "verifier" in the "nfs_client_id4"
  structure. The idea of the verifier is each time the
  client reboots (creates a new incarnation of itself),
  the client produces a new verifier, but keeps the "id"
  field of the nfs_client_id4 structure constant between
  incarnations. The idea was that the server would use
  the verifier field to detect when the client rebooted,
  so that a retried SETCLIENTID due to an RPC timeout did
  not cause the server to trash client state.

- SETCLIENTID_CONFIRM was not in the original draft
  specification.  I believe it was David Robinson's
  comments about guarding from the "Byzantine router"
  that caused me to propose SETCLIENTID_CONFIRM (and I'm
  assuming I introduced SETCLIENTID_CONFIRM ... if not
  apologies to whoever credit is due). The scenario is
  a SETCLIENTID is done at time T, it times out because
  of a router problem, and the client resends at T+1. At
  T+100 the client has established lots of lock state. At
  T+101 the router comes back to life, and transmits
  the SETCLIENTID retry. Server receives it; there is a
  duplicate request cache miss; and the client loses its
  lock state.

  SETCLIENTID_CONFIRM protects the client because the
  server does not trash lock state from a SETCLIENTID
  unless a SETCLIENTID_CONFIRM follows. Since the
  SETCLIENTID_CONFIRM's arguments rely on the results of
  the previous SETCLIENTID, it is very improbable, if not
  impossible, for the Byzantine router to mess up the
  client's state.

- By the way, the principal used for SETCLIENTID must match
  that used for the corresponding SETCLIENTID_CONFIRM.

- With the introduction of SETCLIENTID_CONFIRM, the
  verifier field for the purpose of detecting client
  incarnation changes is not needed, but read the next
  two points before rushing to update your .x file.

- The inclusion of the callback and callback_ident fields
  in hindsight, may have been a mistake. The reason is that
  SETCLIENTID ought to be focused on the tricky problem
  of safely establishing the clientid4 value. There should
  have been a separate, SETCALLBACK operation. This way, we
  would not issues with how to change callback information
  without perturbing our hard won clientid4 value. I am
  not proposing adding the SETCALLBACK operation though,
  because it is not necessary (IMHO) to make the protocol
  work.

- Although nfs_client_id4 no longer needs the verifier
  field for protecting from the Byzantine router destroying
  a client's lock state, Dave suggested to Spencer
  and me to continue to use the verifier but employ
  it for updates of the callback and callback_ident
  fields. I.e. if the client incarnation verifier in
  SETCLIENTID's arguments is that same as what the server
  has previously recorded, then it treats the call as an
  update to the callback information. It then returns
  the old clientid4 value.  The update of the callback
  information MUST be confirmed with SETCLIENTID_CONFIRM
  before the server commits the changes.  This is to
  protect from the Byzantine router updating the server
  with stale callback information. While the protection
  is not as iron clad as how the client's lock state is
  protected, it is pretty good. If pretty good is not
  good enough, then bring back the setclientid_confirm
  construct from RFC3010 (I go away for two years, and
  you guys change things behind my back. See what happens).

Let's look at the known issues, and resolve them per the
framework described above.

From Spencer's list of unresolved issues from nfsv4.org:

- Issue85: Clarify that SETCLIENTID/SETCLIENTID_CONFIRM
  sequence with same verifier is a no-op (no release
  of state).

  ---

  mre: Yes there is no release of state in that event. But
  the callback information is updated (assuming it is different;
  the protocol specification does not care if the server does a
  comparison before overwriting or not).

- Issue86: For delegation recall, the client needs to
  use separate programs or port numbers to differentiate
  between servers callbacks since filehandles and stateids
  may collide between servers

  ---

  mre: The SETCLIENT arguments include callback fields
  (callback and callback_ident) that the client should
  collectively make unique for each server it talks to.
  Its OK, and expected that the "callback" field will be
  the same for each server, but if so, the callback_ident
  field needs to be unique.

- Issue91: Clarify that the server is supposed to
  free state for a restarted/rebooted client when
  SETCLIENTID_CONFIRM is received. The document is
  inconsistent, sections 8.1.1 and 14.2.34 say that the
  server should wait for SETCLIENTID_CONFIRM; Section
  14.2.33 says the server can free state upon receipt
  of SETCLIENTID.

  ---

  mre: Server cannot free any state based on a SETCLIENTID.
  Only a SETCLIENTID_CONFIRM call free state, and even
  then only one that is called with a clientid4 that has
  NOT PREVIOUSLY BEEN CONFIRMED.

- Issue131: Clarify when the lease period starts at the
  server (SETCLIENTID).

  ---

  mre: The lease period, cannot start with a SETCLIENTID
  or a SETCLIENTID_CONFIRM. The lease period starts with
  the first OPEN.

- Issue175: Should SETCLIENTID with same client identifier,
  same verifier and a different callback program should
  return an error (not clear client state).

  ---

  mre: SETCLIENTID MUST never clear client state under
  any circumstances.  A SETCLIENTID with the same client
  identifier, verifier, and different callback information
  will start the process of updating the callback
  information, and return the old clientid4 value (and if
  we decide to bring it back, a fresh setclientid_confirm),
  but only if it is followed by a SETCLIENTID_CONFIRM.

Now the client id issues raised by Rick, Peter, and others
in July.

** On July 8, Rick asks

> How is the server supposed to distinguish between a client
> doing a retry of setclientid and different clients doing

Let x be the "id" field of the nfs_client_id4 struct,
v be the verifier, and c be the clientid4 value.

A client that does a retry of a SETCLIENTID { v, x }
gets the following result depending on the following cases:

- duplicate request cache (DRC) hit: return the result
  cached in the the DRC.

- DRC miss and server has recorded a confirmed { v, x, c }
  (i.e. the client previously did a proper
  SETCLIENTID/SETCLIENTID_CONFIRM).  Since the verifier
  matches, the server treats this as possible callback
  information update and returns c. The update is only
  confirmed if followed up by a SETCLIENTID_CONFIRM
  { c }.  It's possible this request is one from the
  Byzantine router that has stale callback information.
  Not a big problem; unless the client follows up with
  SETCLIENTID_CONFIRM, it won't clobber the callback
  information. If the retry has the same { v, x } and
  callback information, then absolutely no harm done.
  In any case, the server responds with the old clientid4
  value c.

- DRC miss and server has recorded a confirmed { u, x, c }
  such that v != u and has not recorded any unconfirmed
  { *, x, * } record for x.  The server records an
  unconfirmed { v, x, d } (d != c). The server awaits
  confirmation of d via SETCLIENTID_CONFIRM.

- DRC miss and server has recorded a confirmed { u, x, c }
  such that v != u and recorded an unconfirmed { w, x,
  d } such that c != d. Whether w == v or w != v makes no
  difference.  The server simply tosses the unconfirmed {
  w, x, d } and replaces it with an unconfirmed { v, x, e }
  (e != d, e != c). The server awaits confirmation of e
  via SETCLIENTID_CONFIRM.

- DRC miss and server has no confirmed { *, x, * } for
  x. It may or may not have recorded an unconfirmed { u,
  x, c }.  Any unconfirmed record { u, x, c }  regardless
  whether u == v is replaced with an unconfirmed record {
  v, x, d } where d != c.

Let's finish this off and discuss retried
SETCLIENTID_CONFIRM requests. The clientid4 value is c.

- DRC hit. Return the cached result.

- DRC miss, and the server has recorded a confirmed { *, *, c }.
  Return NFS4ERR_OK if principal's match.

- DRC miss, and the server has not recorded a confirmed { *, *, c },
  and has recorded an unconfirmed { *, *, c }. This means the client's
  first SETCLIENTID_CONFIRM attempt was dropped. Server doesn't know
  this is a retry, so it processes it as if were a first try.

- DRC miss, and the server has no record of a confirmed or
  unconfirmed { *, *, c }.  Return NFS4ERR_STALE_CLIENTID. 

> a setclientid with the same "id". (The latter is supposed
> to get the response NFS4ERR_CLID_INUSE whereas the former
> should get an NFS_OK with the clientid?)

If the principals do not match, then NFS4ERR_CLID_INUSE is
the correct error to return. But that is an issue orthogonal
to retries.

> It looks like the Univ. of Michigan server differentiates
> between these two cases by looking at the source IP#
> (not in the callback info in the request, but from the
> transport layer). Does this sound right, or should it use
> the callback net address info or ??
>
> The above is referring to the case where both the "id"
> and "verifier" match one that is already in the server's
> list.

Use of the callback information for determining whether
to allow the establishment of a clientid4 and possible
termination of client state compounds the original sin
of not having a separate SETCALLBACKINFO operation from
SETCLIENTID. As Carl B said on July 18, "the callback
information should be thought of as an attribute of the
client, not part of its unique signature".

What you describe as UMich doing doesn't sound right.
The same verifier and id could be used to set new
callback information. The same client could have multiple
interfaces, and so its source address could change
between resets of callback information. Use the principal
comparison and not the source IP address check to decide
whether to return NFS4ERR_CLID_INUSE.

Since there is the practical problem of client's sharing
the principal, most likely with AUTH_SYS, and most
unavoidably with AUTH_NONE, clients that share principals
need to select a unique id string that includes something
other than, or in addition to the IP address. The FQDN
is not sufficient for that additional something, because
the client could be using DHCP to assign its IP address,
and so its FQDN changes with the IP address. Similarly,
the AUTH_SYS machine name is not sufficient, because it
is often derived from the FQDN, which is derived from the
IP address, which can change.

It would be best if the NFSv4 client implementation
made few assumptions about when and how the IP address
changes. So, including in the unique id string something
that is really hard to conflict, would be best.

So in addition to the server's IP address (See Issue 111),
the client's IP address and the FQDN, the client could
also include stuff like a MAC address, the machine serial
number (for privacy reasons, best to XOR this with some
of the other components), the timestamp of when the NFSv4
software was installed, a digital signature, and/or a true
random number at least 32 bits wide. What it includes is
up to the implementation. But it SHOULD pick something that
will be unique and not tied the client's address or name.

Reading the specification, there are lots of places
where it is not clear that principal check is the deciding
factor as to whether NFS4ERR_CLID_INUSE should be returned.
But that is the only check that will accomodate all the
operating environments.

That said, there will inevitably be NFS4ERR_CLID_INUSE
errors reported whether due to server or client bugs, or the
improbable occurring.  As Spencer wrote on July 18, the
client that has done its level best to pick a collision
free id string that still gets NFS4ERR_CLID_INUSE can use
this algorithm: "have it [the client] wait a server lease
period and retry the operation". I would add, that if the
client still gets NFS4ERR_CLID_INUSE, we can assume that
another client has usurped the id string. This is why
the SETCLIENTID4 results look like this:

  union SETCLIENTID4res switch (nfsstat4 status) {
   case NFS4_OK:
	   SETCLIENTID4resok      resok4;
   case NFS4ERR_CLID_INUSE:
	   clientaddr4    client_using;
   default:
	   void;
  };

The client that gets an NFS4ERR_CLID_INUSE after waiting
the lease period should log the offending client's
address. Such a client might also check whether the
offending address is one that of the client getting the
NFS4ERR_CLID_INUSE error.

Short of implementing Carl B's server assigned id protocol
(which I'm not sure can be made to work), I don't know
what else to do. An attribute of NFS is that the server
is stupid and the client is smart. If the client is as
stupid as the server, it is not the server's problem.

I think my response and rest of my message deals with
Peter A's and Dave N's responses to Rick's question.

** On July 15, Rick wrote:

> So, the only really safe thing for the server to do is to
> reply CLID_INUSE whenever the same nfs_client_id4.id as
> a current one shows up, BUT...
>
> --> then clients get CLID_INUSE after a reboot or
>     retransmit of a setclientid op rpc and the server
>     can't throw away the old clientid state until the
>     lease is up.

If the principal and client id string matches, and if
the verifier is different, the SETCLIENTID followed by
the SETCLIENTID_CONFIRM is all the trigger the server
needs. Returning CLID_INUSE is unnecessary and not right.

> I think, for now, my code is going to paranoid and reply
> CLID_INUSE unless all of, the same principal/user, callback
> address and return IP# are the same as what was given
> in the setclientid request that allocated the clientid.
> (This way, the verifier still does tell the server about
> a client reboot/retransmit, but minimizes the chance of
> "musical chairs" if two clients do somehow get the same
> nfs_client_id4.id.)

If clients picked ids per my suggestions previously,
this won't be a problem 99.99% of the time.

** On July 19, Peter A wrote:

> > > This nfs_client_id4.id looks independent
> > > of the IP address to me. And your
> > > server returns CLID_INUSE if the client
> > > IP changes? Then, how do you
> > > change IP of the client, without getting
> > > CLID_INUSE?
> >
> > It doesn't.  The client will receive CLID_INUSE
> > and in the current implementation will error out of
> > the operation (I will be opening a bug on the Solaris
> > code to have it wait a server lease period and
> > retry the operation).
>
> If your Solaris server releases the clientid after a
> lease period, then this will work fine.  But according
> to the current spec, servers are not required to do so.
> Servers can keep the clientid forever, which means the
> client would retry and get CLID_INUSE forever.  Or at least
> until some administrator manually clears the clientid,
> if this is possible with the current server.

This is section 8.1.2, Server Release of Clientid.

I suggest adding this to section 8.1.2:

        Note that if id string in a SETCLIENTID request is
        properly constructed, and if the client takes care
        to use the same principal for each successive use
        of SETCLIENTID, then, barring an active denial of
        service attack, NFS4ERR_CLID_INUSE should never
        be returned.

        However, client bugs, server bugs, or perhaps a
        deliberate change of the principal owner of of
        the id string (such as the case of a client that
        changes security flavors, and under the new flavor,
        there is no mapping to the previous owner) will
        in rare cases result in NFS4ERR_CLID_INUSE.

        In that event, when the server gets a SETCLIENTID
        for a client id that currently has no state, or it
        has state, but the lease has expired, rather than
        returning NFS4ERR_CLID_INUSE, the server MUST allow
        the SETCLIENTID, and confirm the new client id if
        followed by the appropriate SETCLIENTID_CONFIRM.

** On July 19, Peter A wrote:

> I'm still not convinced that the current way of doing
> SETCLIENTID is optimal.  But, since we have much to do,
> I will stop complaining about it, at least until 4.0 is
> finished :-)

One of the things I was adamant about was that the server
should not have to keep a record in non-volatile storage
of the list of clients that held state on it so that it
can visit each client after server reboot. We've seen
with NLM/Status Monitor for NFSv[23] all the problems
with that. Do you use a host name, FQDN, what if there
is no DNS, what if the DNS resoution on the client and
server mismatch (something I ran into this week at the
workplace), IP addresses (what if the client is DHCP-based
or has multiple interfaces, and the interface of record is
removed), etc? And maybe it isn't convenient to maintain
the non volatile storage, or maybe the non volatile storage
gets corrupted.

I believe if you go with a server record model, you can
then go with with a model that lets the server select
the unique identifier, per Carl B's earlier message. But
we don't have a server record model, for which I'm very
grateful.

---

From: Peter Astrand <[email-masked]>
Date: Mon, 22 Jul 2002 14:44:33 +0200 (CEST)

> I'll try to clean it up (with some help previously from

Thanks for these detailed explanations!

>   without perturbing our hard won clientid4 value. 

This is a very good observation. 

>I am
...
> [9 examples of different cases]

You are right; it's probably not necessary to add a SETCALLBACK op to make
the protocol work.  However, to me, the SETCLIENTID op has turned very
complicated.  I am positive to introducing a SETCALLBACK operation, as
well as removing the verifier from nfs_client_id4 (but I do understand
that this may not be possible at this stage).  Maybe something like this:

 typedef opaque nfs_client_id4<NFS4_OPAQUE_LIMIT>;

 struct SETCLIENTID4args {
             nfs_client_id4  client;
     };

(Must be followed by SETCLIENTID_CONFIRM to release state)

 struct SETCLIENTID_CONFIRM4args {
             clientid4       clientid;
     };

 struct SETCALLBACK4args {
             clientid4       clientid;
             cb_client4      callback;
             uint32_t        callback_ident;
     };

(Must be followed by SETCLIENTID_CONFIRM to update cb info)

> whether to return NFS4ERR_CLID_INUSE.
...
> But that is the only check that will accomodate all the

Yes, this is my opinion as well. I started to think I was the only one
with this opinion... :-)

(Btw, I think it's a bit odd that we must deal with low-level problems
like Byzantine routers in a high-level operation like SETCLIENTID.  
Currently, we have several different ways of protecting ourselves from
re-sent packages and Byzantine routers: Duplicate Request Cache (which is
far from perfect), sequence-id's, _CONFIRM-operations and verifiers.  Why
not use a single mechanism for the entire v4 protocol?)

-- 
/Peter Åstrand <[email-masked]>