Subject: draft-ietf-dhc-failover-09.txt

From: Kim Kinnear <[email-masked]>
Date: Fri, 20 Jul 2001 15:00:56 -0400

Folks,

I've just submitted the latest version of the failover draft,
draft-ietf-dhc-failover-09.txt.

It has a number of changes in it, derived both from discussions
held during the last IETF in Minneapolis in December of 2000, as
well as some discussions which were generated by problems
discussed on this list.

I've given a complete list of the changes below.  All are fairly
minor, but several are more than editoral corrections or
clarifications (of which there were also plenty).

Thanks to Bernie for another *stellar* editorial review which
added many clarifications, and thanks to Ted bringing up some
real-world issues that are now handled acceptably.  (I hope you
agree, Ted, since I believe I took every one of your
suggestions.)

In talking with Ralph, we believe that it makes sense to put the
failover draft through another WG last call immediately after
London IETF, and then (if possible), move quickly to IESG last
call.

At least that is the plan.

As I won't be able to attend London IETF, there is no need to
hold your comments until then -- please bring any problems you
have with the -09 version of the draft up on this list.

Cheers -- Kim

==============================================

Here is a SUMMARY of changes made to the -08 version of the draft to
yield the -09 version of the draft:

1.  Fix a potentially major inefficiency in recovery (which has
actually occurred moderately often in production use) by adding a
new failover state: RECOVER-WAIT.

2.  Fix a problem which occurs when two existing servers, both
with active lease state, become part of a failover pair.

3.  Clarification on reusing lease when in PARTNER-DOWN state.

4.  Clean up some security Issues:

5.  Fix Bootp problem with communications interrupted:

6.  Server-id option removed, relationship-name option added.

7.  Textual and editoral corrections from Bernie Volz, Ted Lemon,
and Kim Kinnear.

--------------------------------------------

Here are the changes IN DETAIL:

--------------------------------------------

1.  Fix a potentially major inefficiency in recovery (which has
actually occurred moderately often in production use) by adding a
new failover state:

The problem:

    It turns out that there's a problem in the recovery process.
    Let's say I go into RECOVER, and send an UPDREQALL.  I get
    all the updates, and ack them, and then I get the UPDDONE.
    Now I set a timer for start time of state (STOS)+MCLT.  When
    that goes off, I make the transition into RECOVER-DONE.  The
    peer makes the transition into NORMAL, and then I make the
    transition into NORMAL.

    But what happens if I get interrupted after I've set that
    timer, but before it's expired?  Currently, I come back in
    the RECOVER state, with no knowledge that I had gotten a
    complete update.

In the -08 draft, this happens:

    So I guess I send another UPDREQALL, and then things proceed
    normally.  But this can be a pretty expensive thing to do,
    and this is not a very far-fetched scenario.

The solution to this problem:

Add an additional state, RECOVER-WAIT, which we enter when we get
the UPDDONE from the partner.  When we enter RECOVER-WAIT, we
start a timer which should expire at the time equal to when the
server went down plus the MCLT or later.  When this timer
expires, we move into RECOVER-DONE state.

-----------------------------------------------------------------

2.  Fix a problem which occurs when two existing servers, both
with active lease state, become part of a failover pair.

The problem:

    There is a problem if two failover peers come up and neither
    one has done failover before but both peers have valid lease
    state at that time.   In this case, they each need to see all
    of their peers' non-virgin leases.   If the primary starts in
    PARTNER-DOWN and the secondary in RECOVER (as written in the
    -08 draft), then the primary may hand out leases that the
    secondary already assigned back when both servers were
    operating independently.

The solution:

    Start both primary and secondary servers in RECOVER. Both
    peers send UPDREQALL messages at the same time.  There is a
    slight change: if I am in RECOVER-DONE and the peer goes into
    RECOVER-DONE, I have to go into NORMAL.

    An implementation SHOULD have a configuration parameter which
    can be set on the primary to indicate that the primary should
    move to PARTNER-DOWN state if has never had any contact with
    the secondary server and is unable to contact that server on
    startup.  This will allow a server which is made a failover
    server to continue to operate until it can rendevous with the
    newly created secondary.  It can only be used in cases where
    the secondary has no existing lease state information.

-----------------------------------------------------------------

3.  Clarification on reusing leases when in PARTNER-DOWN state.

>From the -08 draft:

   In addition, once a server has transitioned to PARTNER-DOWN
   state, it MUST NOT reallocate an IP address from one client to
   another client until an additional MCLT interval after the
   lease by the original client expires.  (Actually, until the
   maximum client lead time after what it believes to be the
   lease expiration time of the client.)

The problem here is that if the peers were in communications-
interrupted for an extended period of time, the secondary may
have renewed the lease to MCLT quite a few times, so that the
actual lease expiry time on the client is well past the lease
time recorded on the server that is now in partner-down state.
To be safe, I think that this lease should not be reallocated
until partner-down-stos + MCLT.

Make it clear that this must be the greater of the time mentioned
above or the start-time-of-state+mclt.

-------------------------------------------------

4.  Clean up some security Issues:

a.)  To look into the MD5/HMAC issue.  Bernie decided we should
use HMAC.

b.) We will require that the message digest option is first in
the packet if you are doing a message digest.

c.) We will specify that you must keep the last time a connect
message came in (the time in the message), and if we ever saw
that connect message or one earlier then we would reject the
connect.

d.) We will require that the XID's must go up, and that you must
reject a message that didn't have an increasing XID.

e.) We will suggest a configurable parameter for time
differential when using message digests.

--------------------------------------------------

5.  Fix Bootp problem with communications interrupted:

The problem is:  If there are two server in
communications-interrupted that are both responsive to dynamic
BOOTP, then the same client could get an address from both of
them.

There are two possible solutions:

	1.  When the server reconnect and find the conflict, they
	should not take the "primary wins" strategy, but rather
	agree to disagree.  The client will keep both addresses,
	since neither server can know which one he remembers.
	Requires manual intervention to fix.

	2.  When in communications-interrupted state, a server
	doesn't become responsive until it is in partner down
	state.  This would cause a lack of responsiveness or
	dynamic bootp clients, but would keep them from ever
	getting into this situation.

We will require solution #1 in the draft.

-----------------------------------------------------

6.  Server-id option removed, relationship-name option added.

On the failover server-id option, we will put a string valued
name (the relationship-name option) in the packet which would
identify the relationship.  This would allow more than one
relationship between two machines using the same IP address
(instead of the previous requirement for multihoming to do this).

It will also allow for moving a relationship more easily.  You
may choose to allow configuration of the IP address of the
partner, and if they connect from a different place, you can
choose to use it or not, but you don't have to.

If no security, MAY want to validate IP of partner instead of
just allowing anyone with proper string to operate.  Can use the
source-ip of the TCP connection to do that.

The relationship-name option is in the connect message.

We will take out the server-id option.

Because of this, the connect message processing is different --
the connect processing must  look at the relationship-name as
opposed to the sending-server-ip-address option.

---------------------------------------------------------

7.  Textual and editorial changes.

a.) Ted's "has transitioned" -> "has made the transition to".

b.) Bernie's -08 review.  Moderate editorial changes and
clarifications.  Lots of actual changes.

c.) updated bibliography for draft->rfc progress