Subject: TCP for failover protocol

From: Kim Kinnear <[email-masked]>
Date: Wed, 7 Jul 1999 17:46:25 -0400 (EDT)

Folks,

As you no doubt noticed, there is a new version of the failover draft,
which I put together with the help of Bernie Volz and Steve Gonzci:

	draft-ietf-dhc-failover-04.txt

I wanted to continue a discussion that Dave Erdmann started a while back
about using only a single TCP connection for the failover protocol,
instead of using a TCP connection for bulk data transfer *and* a UDP
connection for sensing whether one can communicate with the partner
server.

Ted Lemon started the drive for this during the failover meeting last
February.

Well, the -04 draft indeed is defined with only one TCP connection and
no UDP connection, so in that sense I believe that I've achieved the
goal.  However, as is so common, there are tradeoffs.

Let me quote from the draft and add some words to try to explain what I
did and why, and generate some discussion on it prior to Oslo.

First, let me set the stage a little bit.

There are two reasons a failover server needs a connection to its
failover partner:

    1. To pass binding database updates between them.

    2. To determine if the partner is able to communicate with it.

One of the slightly unusual characteristics of the failover protocol is
that, because of the "lazy update" nature of the dataflow in #1, it is
not a particularly time critical dataflow.  The long term throughtput is
important, but the latency of individual messages is not.  In
particular, on the receiving server, processing DHCP client requests is
considerably more important than processing failover binding database
updates.    Note also that as part of initialization or during recovery,
a server may have 10,000 or 20,000 BNDUPD messages to send to its
partner, which may take many minutes to processes.  While large, these
aren't contrived situations, but real world examples.

However, the sensing and detection required by #2 is moderately time
critical, in that a server that doesn't know that the other server is
not responding will not itself respond to the correct DHCP client
requests (i.e., DHCPDISCOVERS), and new clients will not be able to get
IP addresses if one server doesn't know that its partner is down or
disconnected.

TCP is well suited to use #1 by just passing data down the wire and
processing it as fast as possible (but with less priority than DHCP
client requests on the receiving server).  However, if the TCP
connection is being used in this way, there are problems in using it for
the sensing and detection function required by #2, above.

Below I quote from section 3.5 of the most recent failover draft:

-------------------------------------------

3.5.  Using TCP to detect partner server failure

   There are several characteristics of TCP that are important to the
   functioning of the failover protocol, which uses one TCP connection
   for both bulk data transfer as well as to assess communications
   integrity with the other server.  Reliable and ordered message
   delivery are chief among these important characteristics.

   It would be nice to use the capabilities built in to TCP to allow it
   to determine if communications integrity exists to the failover
   partner but this strategy contains some problems which require
   analysis.  There exist three fundamental cases for an open TCP con-
   nection that must be examined.

      1.  When no data is being sent then no messages are traveling
          across the TCP connection.

      2.  When data is queued to be sent, and the receiver has not
          blocked the sending of additional data, then messages are
          flowing across the TCP connection containing the applications
          data.

      3.  When data is queued to be sent, and the receiver has blocked
          the transmission of additional data, then persist messages are
          flowing from the receiver to the sender to ensure that the
          sender doesn't miss the receiver opening the window for
          further transmissions.

   The first case can be turned into the second case by sending
   application-level keep-alive messages periodically when there is no
   other data queued to be sent.  Note TCP keep-alive messages might be
   used as well, but they present additional problems.
   [And are not further considered here -- we can discuss them if
   necessary.]

   Thus, we can ensure that the TCP connection has messages flowing
   periodically across the connection fairly easily.  The question
   remains as to what TCP will do if the other end of the connection
   fails to respond (either because of network partition or because the
   receiving server crashes). TCP will attempt to retransmit a message
   with an exponential backoff, and will eventually timeout that
   retransmission.  However, the length of that timeout cannot, in gen-
   eral, be set on a per-connection basis, and is frequently as long as
   nine minutes, though in some cases it may be as short as two minutes.
   One some systems it can be set system-wide, while on some systems it
   cannot be changed at all.

   A value for this timeout that would be appropriate for the failover
   protocol, say less than 1 minute, could have unpleasant side-effects
   on other applications running on the same server, assuming that it
   could be changed at all on the host operating system.

   Nine minutes is a long time for the DHCP service to be unavailable to
   any new clients that were being served by the server which has
   crashed, when there is another server running that could respond to
   them immediately as soon as it determines that its partner is not
   operational.

   The conclusion drawn from this analysis is that TCP provides very
   useful support for the failover protocol in the areas of reliable and
   ordered message delivery, but cannot by itself be relied upon to
   detect partner server failure in a fashion acceptable to the needs of
   the failover protocol.  Additional failover protocol capabilities
   will need to be created to support timely detection of partner server
   failure.  See section 8.3 for details on this mechanism.

-------------------------------------------

This explains the problem.   

Here is the solution that is contained in the draft:

---------------------------------------------

8.3.  Using the TCP connection for determining communications status

   The TCP connection is used to determine the communications status of
   the other server, i.e., communications-ok, or communications-
   interrupted.

   Three things must happen for a server to consider that communications
   are ok with respect to another server:

      1.  A TCP connection must be established to the other server.

      2.  A CONNECT message must be received and a CONNECTACK message
          sent in response.  The CONNECT message is used to determine
          the identify of the failover endpoint of the other end of the
          TCP connection -- without it, the failover endpoint cannot be
          uniquely determined.  Without knowledge of the failover end-
          point, then the entity with which communications is ok is
          undetermined.

      3.  A STATE message must be received from the other server over
          the connection.  This STATE message initializes important
          information necessary to the operation of the state machine
          the governs the behavior of this failover endpoint.

   There are two ways that a server can determine that communications
   has failed:

      1.  The TCP connection can go down, yielding an error when
          attempting to send a message.  This will happen at least as
          often as the period of the tSend timer.

      2.  The tReceive timer can expire.

   In either of these cases, communications is considered interrupted.

   Several difficulties arise when trying to use one TCP connection for
   both bulk data transfer as well as to sense the communications status
   of the other server.   One aspect of the problem stems from the dif-
   ferent requirements of both uses.  The bulk data transfer is of
   course critically important to the protocol, but the speed with which
   it is processed is not terribly significant.  It might well be
   minutes before a BNDUPD message is processed, and while not optimal,
   such an occasional delay doesn't compromise the correctness of the
   protocol. However, the speed with which one server detects the other
   server is up (or, more importantly, down) is more highly constrained.
   Generally one server should be able to detect that the other server
   is not communicating within a minute or less.

   These differing time constraints makes it difficult to use the same
   TCP connection for data transfer as well as to sense communications
   integrity.   See section 3.5 for additional details on TCP.

   The solution to this problem is to require a that some message be
   received by each end of the connection within a limited time or that
   the connection will be considered down.  If no messages have been
   sent recently, then a CONTACT message is sent.

   In the case where there is no data queued to be sent, this is not a
   problem, but in the case where there is data queued to be sent to the
   partner, then the CONTACT message will not actually be transmitted
   until the queued data is sent.  Section 3.5 explains why waiting for
   TCP to determine that the connection is down is not acceptable, and
   leads a requirement that the receiving server never block the sending
   server from sending CONTACT packets.

   In order to meet this requirement, each server tells the other server
   the number of outstanding BNDUPD messages that it will accept.  The
   receiving server is required to always be able to accept that many
   BNDUPD messages off of the connection's input queue even if it cannot
   process them immediately, and to accept all other messages immedi-
   ately.

   Thus, the sending server's TCP is never blocked from sending a mes-
   sage except for very short periods, less than a few seconds unless
   the network connection itself has problems.  In this case, if the
   CONTACT messages don't make it to the partner then the partner will
   close the connection.

8.4.  Using the TCP connection for binding data

   Binding data, in the form of BNDUPD messages and BNDACK messages to
   respond to them, are sent across the TCP connection.

   In order to support timely detection of any failure in the partner
   server, the TCP connection MUST NOT block for more than a very short
   time, on the order of a few seconds.  Therefore, a server that is
   sending BNDUPD messages MUST send only a restricted number before
   receiving BNDACK messages about previous messages sent.

   The number of outstanding BNDUPD messages that each server will
   accept without causing TCP to block transmission of additional data
   (i.e, CONTACT messages) is sent by each server in the CONNECT and
   CONNECTACK messages in the max-unacked-bndupd option.

8.5.  Using the TCP connection for control messages

   The TCP connection is used for control messages: POOLREQ, UPDREQ,
   STATE, UPDREQALL and the corresponding reply messages: POOLRESP,
   UPDDONE.  A server MUST immediately accept all of these messages from
   the TCP connection.  A server MUST immediately accept any BNDACK
   which is received as well.

8.6.  Losing the TCP connection

   When the TCP connection is lost, then communications is not ok with
   the other server.  A server which has lost communications SHOULD
   immediately attempt to reconnect to the other server, and should
   retry these connection attempts periodically.

   Any BNDUPD or other messages that have been received but not yet pro-
   cessed from the partner SHOULD be processed as soon as possible.

-------------------------------------

Thus, the solution documented above, is for the servers to agree on the
number of un-acked binding update messages that they can *always*
accept, and then for one server to never send more than that to its
partner.  In this way, the TCP connection is always kept open for
control messages and for CONTACT messages.  Thus each server will
receive a message (and be *able* to receive a message) of some type
within every few 10's of seconds -- usually either a BNDUPD or CONTACT
message, but it could be any message.  And because the TCP connection is
kept "open", that message will always go through, and the receiving
server will never be flooded with 10,000 or 20,000 BNDUPD messages.

I expect we'll be discussing this a lot over the coming months, and I'm
planning on hosting a conference call at some time after Oslo to discuss
this (and other) issues in the failover protocol.

Cheers - Kim

---

From: "Bernard Aboba" <[email-masked]>
Date: Thu, 8 Jul 1999 03:05:23 -0400 (EDT)

Overall, I think that this is a good solution to the problem.
As you state, the keepalive messages are sent if no other
data is being sent in order to keep the connection up. I
would not attempt to use TCP keepalive for this since it
was created for quite a different purpose. 

I would also note that there is no need to wait until 
TCP times out in order to conclude that a connection is
down. It is possible to set your own timer for a shorter
value and close the connection if nothing is received up
to that point. Of course, you don't want to set the timer
too short or you will close connections unnecessarily.

---

From: Kim Kinnear <[email-masked]>
Date: Fri, 9 Jul 1999 10:57:23 -0400 (EDT)

>was created for quite a different purpose. 

	Thanks!
>too short or you will close connections unnecessarily. 
	Indeed, the value of tReceive is not specified
	in the draft because it is installation dependent,
	but it is both independent of the TCP timeout
	and, in all cases I can imagine, considerably shorter.

	In a LAN environment (i.e., not a situation
	where servers are separated by a 56K link) I would
	expect to configure tReceive to be 30 seconds and
	tSend to be 10 seconds.  Thus a message would be 
	sent every 10 seconds from each server to the other,
	and if either server didn't receive a message for
	30 seconds, it would drop the connection and assume
	that communications with the other server was
	interrupted.

	I don't think I'd want to configure them much 
	shorter, but then I don't think I'd want to configure
	them much longer either.  But your mileage may vary.

	Cheers -- Kim