Subject: Delegation callbacks proposal

From: Brent <[email-masked]>
Date: Fri, 31 Aug 2001 16:14:51 -0700

An unusual feature of the NFSv4 protocol is that it uses a "callback"
protocol, NFS4_CALLBACK that allows the server to recall a file
delegation asynchronously. It implies that for the purposes of callback,
a client must run an RPC server thread.

The client registers its callback address with the server via the
SETCLIENTID operation. Here's the argument that contains the
address info.
 
        struct cb_client4 {
              u_int cb_program;
              struct clientaddr4 {
                      /* see struct rpcb in RFC 1833 */
                      char *r_netid;  /* network id */
                      char *r_addr;   /* universal address */
              } cb_client4;
      };
 
I was a bit disturbed by the requirement for a netid and
universal address.  To my knowledge, there's no standard
for netid strings.  RFC 1833 says they're just a local
phenomenon, recorded in the netconfig file.  You have to
use the netid to figure out the meaning of the dotted
Universal Address.
 
This registration step, having the client pass its callback
address to the server, will create severe problems on the
Internet.

Whether we like them or not, NATs are showing up all over the place.
I go through two levels of NAT to get to the Internet via
my home DSL line.  Since a NAT hides the client's
real address and port from the server, it means that
an address passed via cb_client4 through a NAT will
be useless to the server.  It's possible that the
growing ubiquity of NATs could render NFSv4 file delegation
useless for most Internet users - the environment where
delegation is most needed.
 
So, I have a suggestion: remove the cb_client4 from SETCLIENTID.
 
Let the server determine the client's address/port from the
incoming SETCLIENTID call and re-use the connection for
the callbacks.  We already do something similar in the
rpcbind protocol - the portmapper's GETADDR call ignores
the client-supplied netid and uses the netid of the 
call message.
 
This is new for RPC implementations, but requires no change to the
RPC protocol. There's nothing in any of the RPC protocol docs that
prevents a client and server from co-existing at each transport endpoint.
An RPC client reading from a socket can tell easily whether
an incoming RPC message is a call or a reply and deal with it
appropriately.
 
There's a bunch of nice features in doing the callbacks this way:
 
        - Connection re-use
 
          It reduces by half the number of TCP connections that
          an NFSv4 server needs to support.  It also eliminates
          connection setup for the callback (unless the connection
          needs to be re-established).
 
        - Works through NATs
 
          The server uses the client's endpoint address (addr/port)
          as mapped by the NAT.  Hence, NATs are transparent.
 
        - Works through firewalls
 
          Most firewalls prevent incoming TCP connections, yet
          permit outgoing connections.  Hence an NFSv4 client
          would be able to set up a connection to a v4 server
          but not receive callbacks.  If callbacks are received
          over the "outgoing" connection, then the client can
          receive callbacks.  
 
        - Transport and address independent
 
          We no longer need to worry about netid or address formats
          in the protocol.  Callbacks will work through the migration
          from IPv4 to IPv6 and should work quite seamlessly even with
          NFS/RDMA.

It does mean a bit of work at the kernel RPC level in creating an function
that sets up a connection-sharing client/server pair.  But I think
the work should be localized and require no changes to the majority of
the RPC API.

Re-use of the incoming connection for callbacks should not affect the
client or the server's ability to reap idle connections.  If a dropped
connection needs to be re-established, then client or server must make
a best effort to re-establish when needed.  If the server needs to callback,
but cannot reach the client, then the callback lease will eventually expire.
that's what it's there for.

So, I propose  cb_client4 be removed from SETCLIENTID and replaced with
wording in that indicates that the NFS4_CALLBACK server be located at
the client's transport endpoint, i.e. if a TCP connection is used, then
re-use the connection.

        Brent

---

From: Mike Eisler <[email-masked]>
Date: Fri, 31 Aug 2001 17:35:58 -0700

> delegation is most needed.

These issues were discussed when Dave made the intial
delegated caching proposal. See
	http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1999/0571.html

The resolution was that if the server couldn't make the callback, then
it wouldn't grant delegations. NAT has been around for a while, the
WG surely wasn't blind to the fact that it would get in the way.

> that's what it's there for.

While we are revisiting old territory,

(see
	 http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1998/0385.html
which introduces the notion of bi-directional RPC )

I suggest revisiting even older territory in

	http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1998/0382.html

and
	http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1998/0024.html

Rick and Uresh are saying basically the same thing: just as current
dialects of CIFS do today, tie the NFSv4 "lease" state to the
TCP connection. When the connection drops, the state the NFSv4
server has is dropped after some grace period. Unlike
CIFS, during the grace period, the client can re-establish its leases, 
or not. 

> the client's transport endpoint, i.e. if a TCP connection is used, then

And if UDP is not used, what then? (This too was discussed in 1998,
see
	http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/1998/0401.html 
"1998: the forgotten year of NFSv4"), is that with UDP you really
need to have the cb_client4 because NFS/UDP clients routinely
use different source ports on successive RPCs to the same
server, and the source port is routinely used for traffic to
other servers, and also other traffice over other
application protocols. You cannot reserve a UDP port for
one client/server "connection" because you might run out of them,
and in some UDP/IP implementations, one UDP socket won't
bear all the traffic a particular physical medium is capable of;
that's why many NFS clients have pools of client handles
bound to bound UDP sockets.

So, the implication of requiring bi-directional RPC is
no NFSv4/UDP, or at least, no NFSv4/UDP that uses
stateid or seqid generating RPCs.

	-mre