Subject: Minutes of 2nd Failover conference call

From: Kim Kinnear <[email-masked]>
Date: Fri, 8 Oct 1999 12:56:54 -0400 (EDT)

Yesterday, we held the second of three conference
calls concerning the current failover protocol draft:

The third (and last of this series) will be held Thursday, Oct
14, from 12-3pm EDT.  Please contact me if you are interested in
attending.

Yesterday's meeting was well attended, and we made a lot of
progress (again)!  Thanks to all of you who attended this call.
The roster of attendees is at the end of this mail.

Cheers -- Kim

-------------------------------------------------

Minutes from the 10/7/99 Failover conference call.

We had good attendance again at the conference call held on
Thursday, Oct 10, 1999.  The agenda was: conflict resolution,
initialization, and state transitions.  We covered those issues,
as well as covering some additional issues.

IP Address Question
-------------------

While we were waiting for people to arrive at the meeting, Andre
Kostur asked a question in email and Kim tried to answer it.  The
question was: what about a server which determines the addresses
it can give out based on a particular client's identity?

The answer is that the failover protocol tries to stay away from
these issues.  The plan is that each server in the failover pair
is configured identically, and so that based on the IP address,
the receiving server knows into which bucket (pool, scope) an
address falls.  The sending server and receiving server are
supposed to agree, so that the protocol doesn't have to know
about this.  Likewise, the primary server will send to the
secondary server the correct number of addresses in each
"sub-pool" or whatever, and the secondary server will know about
each sub-pool and will put the newly available addresses in the
sub-pool.

Conflict Resolution
-------------------

Kim Kinnear started off with a general discussion of where
conflict resolution is handled within the protocol.  He pointed
out that it isn't only when a server is in POTENTIAL-CONFLICT
state, but rather whenever a server receives a BNDUPD it needs to
determine if it is willing to accept that information from the
other server.  Thus, conflict resolution is an ongoing, real-time
operation.

The two kinds of conflict resolution mentioned by Kim are:

     1.  Where one IP address is leased to one client on one
     server and a different client on another server.

     2.  Where one client has one IP address leased on one server,
     and that same client has a different IP addresses leased on
     the other server.

Steve Gonczi added two more:

     3.  One server shows an IP address as free, and the other
     server shows it as leased.

     4.  One server has a different expiration time than the other
     server.

Kim responded that #3 and #4 were, at least in his opinion, not
conflicts but just normal parts of the way the protocol works.

Kim then pointed out that there are a number of times that are
sent in a BNDUPD:

     lease-expiration-time
     potential-expiration-time
     client-last-transaction-time

and a larger set that must be saved in the stable storage on
each machine:

     lease-expiration-time -- time given to client
     sent-potential-expiration-time -- latest time sent to partner
     acked-potential-expiration-time -- latest time acked by partner 
     received-potential-expiration-time -- latest time received from 
                                           and acked to partner

A server can offer/ack to a DHCP client a time that is no longer
than the MCLT beyond the max( received-potential-expiration-time,
                               acked-potential-expiration-time).

When a server goes into PARTNER-DOWN state, it must wait the
MCLT beyond the max( lease-expiration-time,
                      sent-potential-expiration-time,
		     acked-potential-expiration-time,
		     received-potential-expiration-time ) 
for each IP address.

The reason for this is so that in the simple case, where the
primary updates the secondary and then fails, the secondary can
extend the client's lease using the
received-potential-expiration-time, (which will be long), instead
of just the acked-potential-expiration-time, which will in
general be zero since the secondary has never updated the primary
at all at this point.  The pragmatic difference here is that if
we allow the secondary to use only the acked-potential-expiration-time,
it will only extend the lease by the MCLT, but it we allow it to
use the received-potential-expiration time (i.e., a time that it
has acked to the partner), then it can give out a full lease renewal,
which could make a *big* difference in secondary server load.

The consequences of allowing this optimization -- i.e., the
secondary can use a time that it has acked to the primary as well
as a time that the primary has acked to the secondary when making
a lease, then when we enter PARTNER-DOWN state, we must wait not
only for times that the partner has acked to us and times that we
have acked to the partner, but also times that we have sent to
the partner (whether they were acked or not, since the ACK could
have been lost).

Kim agreed to specify what to do on NAK (send and receive),
whether it should cause an update or what.   Several people
reminded Kim to be careful about NAK storms.

Kim agreed to tie the reject-reason codes into the draft, with
references both ways.

While there was considerable discussion about conflict
resolution, the upshot of the discussion was that Kim would try
to make it clearer in the next draft, and people would try to
read it and make sense of it.

Someone proposed a simple conflict resolution strategy -- if
the state in the incoming BNDUPD is different than the one
you have, take it.  Kim said that such a strategy didn't work
in practice.

Kim expressed a longing for a simple, one paragraph statement of
the conflict resolution strategy, but stated that he had tried
several, and they kept developing special state-specific cases
(as in, current state = x, BNDUPD state = y, then do z).

Initialization
--------------

Bernie Volz suggested that we explain the STARTUP state in
section 5 as well as in section 9, and state the general approach
of waiting to determine if we can talk to the other server before
launching off and serving DHCP clients, and Kim agreed.  Bernie
had some other small changes to make to the STARTUP state.

State Transitions
-----------------

We talked about what the states of a lease mean, and someone
pointed out that the FREE state was misdocumented.  Kim agreed to
fix it, and also to include a state transition diagram for the
failover canonical lease states.  The goal for the lease states
in the failover draft is for them to be general enough to model
any vendor's server.

As for the failover endpoint states, Diana Lane brought up a
concern that two servers in the POTENTIAL-CONFLICT state are
unresponsive to DHCP clients, and if that took a while, it would
be a problem.

Kim pointed out that since the resolution process was automatic,
it in general wouldn't be too long.

She asked what happens if communications is interrupted during
this resolution process, and Kim said that the primary server
would go into PARTNER-DOWN state and the secondary would remain
unresponsive in POTENTIAL-CONFLICT state.  She was (rightly)
concerned about the case where the primary server crashing was
the reason that communications was interrupted.

Steve suggested that we move the servers into
COMMUNICIATIONS-INTERRUPTED state in this situation, and Bernie
and Kim said that you can't do that, since when they talk again
they will move back to NORMAL state.  Bernie said that we needed
a new state, and Kim agreed to put on in, and it won't be called
"POTENTIAL-CONFLICT-COMMUNICATIONS-INTERRUPTED".

We talked about conflict resolution, and how long it would take,
and due in considerable part to Richard Jones and Ralph Droms'
approach, we ended up defining three kinds of conflict
resolution:

     1.  "deal-with-it", automatic, conflict resolution, which is
     what is currently specified in the draft.

     2.  "put-it-off" conflict resolution, where a lease that has
     a significant conflict is ABANDONED and left for a human to
     work out later.

     3.  "interactive" conflict resolution, where a human is
     presented with the conflicts and instructs the servers as to
     how to resolve them.

The draft will be extended to support all three styles of
conflict resolution, with only #1 being a MUST.  That is to say,
all servers MUST be able to support a #1 style conflict
resolution regime, but they MAY also supply a #2 or #3 style as
well.

Kim agreed to mention in the draft that if in PARTNER-DOWN state
you use up your own IP addresses prior to using the partner's,
you stand a better chance of not having conflicts.

Miscellaneous
-------------

1.  Steve brought up an issue he had with expired or freed IP
addresses becoming owned by the primary and having to be
reallocated to the secondary.  He said that if we didn't allow a
server to renew a lease about which it had no information, then
we could restructure the way we handle newly freed IP addresses
so they wouldn't have to cycle through the primary.

Kim said that one reason that we had the ability to renew any IP
address as long as it was available, even if we hadn't any direct
information about that client, was so that if a server crashes
after leasing an IP address to a client and before the BNDUPD,
that client could still get an renewal from the other server.
Steve said that was a marginal case, and couldn't affect more
than 2 or 3 clients in the event of a crash, but that the
optimization that he wanted to do would affect lots of clients.

Bernie pointed out that the ability to renew a lease for a client
a server has not heard of is key to other parts of the protocol,
for instance in the situation where a server crashes, the other
makes a bunch of leases, and then it crashes and the first server
comes back up.  This will work today, and any client that got a
lease from either server will be able to get a renewal.  Steve
said that he withdrew his request in the light of these
difficulties.

-------------

     Bernie brought up some of the issues that he had mailed out.
     Here they are in his words (from his mail) with the
     resolution added in indented text:

----------

2.  MCLT Time.  We should give some guidelines as to how to set
it and perhaps even suggest default(s).  What's the impact of a
short time vs a long time.

     Agreed, it will go in the draft.

3.  Switch to PARTNER-DOWN. The text in Section 5.2.1 (Control of
Lease Time) needs to be enhanced.  The key point when switching
to PARTNER-DOWN mode is that the you need to apply the MCLT time
to anything that happened before entering PARTNER-DOWN. You need
not apply MCLT to anything that happens after entering
PARTNER-DOWN.

     Agreed, Kim will augment section 5 with the key points of
     PARTNER-DOWN state.

4.  I think a section on Startup issues should be included (in
section 5).  For example, I would expect that a server coming up
would take some time to attempt to communicate with the peer
before proceeding with possibly old state information it had.
And, there may be cases (such as after loss of storage), where a
server should just wait until the partner is available before
starting.

     Kim pointed out the STARTUP state explanation in section 9,
     and agreed to add information to section 5 as well.

5.  XIDs.  There are packets that have no acknowledgement (such
as the STATE packets).  Should these have an XID?  What XID
should be used (next number or 0 or ?).  We could just say use
whatever you want, just don't expect it to be ACKed.

     Kim said he would clarify the draft.

6.  GRACE Binding Status and grace expiration time.  I'm still
not 100% clear on these with respect to failover, especially what
happens when a server doesn't have these states (perhaps one
server of the pair has them and the other doesn't).  We either
must require these states or provide details on what a server
that doesn't have them does when it receives BNDUPDs with these
states (and what a server that has them does if the peer doesn't
use these states).

     Kim pointed out that section 7.1.2 has information as to what
     to do if you don't implement one of these states, but that he
     would explain it all in text associated with a state
     transition diagram for the failover canonical lease states.

7.  7.3.2 (Receiving the UPDREQ message).  We need to clarify
what happens to NEW BNDUPDs that happen during the UPDREQ
(UPDREQALL) process, such as if new client leases are honored.
Should they be covered by the UPDDONE or not?  If they are, it
may be impossible to send the UPDDONE if clients are leasing
fast enough so I would think they should NOT be included.

     Kim said that he agreed with Bernie that they should *not* be
     tracked as part of the UPDDONE processing, and that he would
     ensure that the draft said this clearly.

8.  Connection close and severe errors.  I think a section on
this topic might be useful.  Send error if possible, but close
connection if not.

     Kim agreed to add some way to send an error message on the
     close of a connection.

9.  A TCP issue ...  7.9.1 states that "The tSend timer SHOULD be
approximately 20 percent of the time in the receiver-timer
option".  I do think that this might be a bit low.  Especially if
you want the servers to detect the other being down within a
minute - this basically means you need to send every 12 seconds
on the connection.  I'd think something like 50% should be
sufficient.  That gives the network half the time-out time to
deliver the packet and causes a send only every 30 seconds.

I also think that this high rate of activity should be documented
clearly; you won't want to run the Failover protocol over a ISDN
or X.25 line unless you're really willing to pay for it.

I'm also wondering whether one could think of having an "idle"
state.  When a server detects that things have been basically
idle for a while (how long is implementation specific), it sends
a new message to the peer telling it "Hey, we're not doing
anything useful so lets close the connection down for now.  When
you (or I) have something useful to send, we'll first attempt to
reopen the connection." Reason for this is that many office sites
will probably be doing nothing more than exchanging CONTACT
messages from 8PM to 7AM (this depends on lease times, etc).
During this period you might otherwise be sending thousands of
CONTACT messages in each direction.  Of course, this would
require both servers to enter COMMUNICATIONS- INTERRUPTED state
such that they can give out leases.  When a lease is given out,
the communication would be restarted and hence all would resume
normal processing.  Also, a check every so often should be done
to avoid triggering PARTNER-DOWN mode.

The changes to the server logic are basically:
a) Detect idle case (such as <n> CONTACT messages in a row), send idle
notification, set idle flag, close connection.
b) Don't restart connection if idle flag set.
c) Clear idle flag whenever an event happens that requires
communication (to assure connection is reestablished) or after n%
of PARTNER-DOWN time has elapsed.

     We talked at length about this issue, and while at least some
     of the people on the call understood exactly what this would
     be useful for, and that it would be useful in some
     situations, nobody besides Bernie felt that it would be
     useful enough to warrant the extra complexity that would be
     required in both specification as well as implementation.
     Kim agreed to mention it as an "open issue", though since
     agreeing to do so, he's not sure even that is a good idea,
     since it received a pretty firm and complete "no" vote from
     everyone who said anything at the meeting besides Bernie.
     Kim will discuss this with Bernie and resolve it.

------------- end of Bernie's items ------------------------

The attendees at this meeting were:
  
Ralph Droms       Bucknell U.         EDT   [email-masked]
David Erdmann     Nortel Networks     EDT   [email-masked]
Steve Gonczi      Process Software    EDT   [email-masked]
Richard Jones     Checkpoint Software PDT   [email-masked]
Kim Kinnear       Cisco Systems       EDT   [email-masked]
Andre Kostur      Incognito           PDT   [email-masked]
Diana Lane        Wal-Mart Stores     CDT   [email-masked]
Ted Lemon         Internet Engines    EDT   [email-masked]
Thomas Narten     IBM/IETF area dir   EDT   [email-masked]
Greg Rabil        Lucent Technologies EDT   [email-masked]
Mark Stapp        Cisco Systems       EDT   [email-masked]
Rob Stevens       Join Systems        PDT   [email-masked]
Bernie Volz       Process Software    EDT   [email-masked]