Subject: Failover protocol, POOLREQUEST issues

From: Steve Gonczi <[email-masked]>
Date: Wed, 8 Dec 1999 16:41:46 -0500 (EST)

Greetings,

I would like to point out a problem and propose a possible solution to the
primary-backup address management scheme in the latest version of the DHC
Failover protocol, which is IMHO slightly broken.
Comments, questions are most welcome!

How it used to work: 

a) When the primary and backup servers are done synchronizing their
databases,
the backup server sends a POOLREQUEST, asking for addresses to replenish its
pool. The primary typically assigns some and sends them over as binding
updates. 

b) When operating in normal mode, the backup server DOES NOT use up any of
these addresses, as it does not server new clients.

c) In communications interrupted mode, the backup server starts using up its
address pool. 

d) Any addresses it has assigned, and freed ( expired or released) revert
back to the ownership of the primary. This happens because the lease
ownership is expressed merely by setting the lease status to "BACKUP". If it
is anything but "BACKUP" the backup server no longer owns it.

e) When the servers re-connect, a) happens again, so life is good.

What is different in the latest draft:

1) Because of load balancing, the backup server now serves new clients from
its pool during normal operation, hence it continually depletes its address
pool. It is virtually guaranteed to run out of addresses during normal
operation,
unless we take steps to prevent this... 
Why? See paragraph d)..

2) The POOLREQUEST /POOLREPLY exchange is relatively expensive on the
primary server side, because the primary
server has to scan its entire lease database to figure out which addresses
(if any) to transfer. 

3) The POOLREQUEST/POOLREPLY exchange should only be used immediately after
the primary has synchronized its database with the backup, so it has a
complete, up-to-date view of the lease database.

Possible solution: 

I) Both servers should flag and keep track of addresses owned by the backup
(in addition to and beyond the lease being in the "BACKUP" state. If a lease
is assigned, i.e. it is "ACTIVE" it still should retain its flag, marking it
as a "BACKUP" lease in BOTH servers.

II) Whenever the backup server detects an expiration, or release of a lease
it owns, it should set the status back to "BACKUP" not "FREE" and should
send an update to the primary with the lease status set to "BACKUP". 

!!!NOTE!!!
The above has nothing to do with my earlier (foolish) proposal of not
allowing a server renew a lease it did not own, if it was marked
free/backup. 

Another issue, loosely related to the previous one:

A) The POOLREQUEST/POOLREPLY exchange should work as an initial pool
arbitration process (pretty much as it works today). The pool percentage is
either a configuration value or possibly a self-tuning parameter (?)

B) To allow the administrator to tune the backup/primary address ratio (say,
based on server load analysis)
we should allow the primary to TAKE AWAY, as well as hand over addresses. 

C) Hence the number of addresses allocated which we report in the POOLREPLY
should either allowed to be
negative as well as positive, or it should be changed to a mere status value
( 1==success, 0==failure)

D) For B) to work, we must make sure the backup server is not serving
clients until the primary has finished transferring the addresses. This is
because it may decide to take away an address, that the backup  could 
have leased in the meantime.

---

From: Richard Jones <[email-masked]>
Date: Wed, 8 Dec 1999 18:59:38 -0500 (EST)

My first take on this is that the proposal muddies the relationship between
primary and secondary in the failover protocol, all in the service of load
balancing - which I believe we've pretty much decided is peripheral
to the function of failover.

I'm in favor of maintaining the priority of the primary server. I don't think
POOLREQ functions need to be expensive, if the primary is implemented
to know how many addresses the secondary has and how many it's allowed
(or configured) to have. Maybe that's going to be harder to do than I
think ;-).

I always assumed that POOLREQ/POOLRESP exchanges could happen
any time, and that when they did happen, there would be a period of
extra bind updates and then it would be done for a while. But compared
to the rest of the traffic, it doesn't seem that awful, especially since the
folks who have implemented already seem to feel that wire traffic is
nothing compared to other functions (like stable storage).

If we have the secondary change the state of a lease to BACKUP and
update the primary, what happens in the case of a conflict? Maybe we'd
simply need another reject-reason code, but there may be more to it
than that.

All in all, I don't think the protocol as it stands is broken in this respect.

Regards,
Richard

---

From: Steve Gonczi <[email-masked]>
Date: Wed, 8 Dec 1999 19:47:13 -0500 (EST)

Richard,

Thanks for your quick review and response.

>to the function of failover.

I believe (and the protocol authors will probably agree) that making the
protocol 
more symmetrical is a good thing. As I recall, the idea has been around for
a
while. The first step in this direction was to allow the backup server to
renew.
leases while in normal mode.

>I'm in favor of maintaining the priority of the primary server. 

What advantage do you see in doing so?

>I don't think POOLREQ functions need to be expensive, if the primary is
implemented
>(or configured) to have. Maybe that's going to be harder to do than I

I am afraid it is a bit harder. New trends in server operation introduce
complexities like
arbitrary lease pools, (with different settings and privileges), which leads
to the primary 
having to leaf through all of them, to hand over the prescribed percantage
of each.

>nothing compared to other functions (like stable storage).

It all depends on the kinds of trafic you are handling. If you want carrier
class performance, 
wire traffic does become an issue.

>simply need another reject-reason code, but there may be more to it

I would very much like to avoid cycling the backup server's expiring leases
through 
the primary if it can be avoided, for reasons of efficiency, and keeping the
same 
lease with the same server.

If someone can come up with a good reason why we have to do this, I would
very 
much like to hear it.

sG

---

From: Ted Lemon <[email-masked]>
Date: Wed, 8 Dec 1999 19:50:21 -0500 (EST)

> to the function of failover.

Wow, I hadn't decided that.   I think that load balancing makes the
failover protocol much more useful.

Other than that, I didn't see anything wrong with what you said, but I
haven't finished implementing yet, so I reserve the right to comment
further... :')

			       _MelloN_

---

From: Richard Jones <[email-masked]>
Date: Wed, 8 Dec 1999 19:53:17 -0500 (EST)

> failover protocol much more useful.

I didn't mean to speak out of turn. I guess what I was trying to say
was that since we decided that it was appropriate to put the load
balancing functionality in another draft, it was less of a candidate
for driving  protocol details. Sounds like that would not meet with
general agreement.

> haven't finished implementing yet, so I reserve the right to comment

By all means! A few weeks from now I'll probably have some
different ideas, too ;-).

Regards,
Richard

---

From: Ted Lemon <[email-masked]>
Date: Wed, 8 Dec 1999 19:56:01 -0500 (EST)

> I guess what I was trying to say was that since we decided that it
> was appropriate to put the load balancing functionality in another
> draft, it was less of a candidate for driving protocol
> details. Sounds like that would not meet with general agreement.

Nope.   We just moved the technical description of load balancing out
of the draft.   It's still required by the draft, unless I am much
mistaken.   So it's as important as it was before the decision to
remove the details from the failover draft was made.

			       _MelloN_

---

From: Richard Jones <[email-masked]>
Date: Wed, 8 Dec 1999 20:35:22 -0500 (EST)

> leases while in normal mode.

I've heard this in the group's discussions, and in general I'm right with it.
But I  think it's fairly safe to say that this protocol can't be completely
symmetrical, so there may be a point where we have to ask, "How
symmetrical is too much? When does complexity overcome the
advantages of symmetry?" I don't really know how to answer that
at this point, and in fact I'm ready to defer to those folks who have
been working with this for much longer than I. It just seems like a
question worth asking, often ;-).

> What advantage do you see in doing so?

Perhaps this is just a reflection of gaps in my experience, but it
seems like maintaining the priority of one over the other reduces
the possibility of conflict in general, and makes arbitration simpler.

> having to leaf through all of them, to hand over the prescribed percantage

I'm not familiar with arbitrary lease pools. Are they analogous to vendor-
or user-classes? Anyway, I can't defend this point strongly because I
haven't done it yet, but yes, it could (make that *will*) certainly get
complicated.

> wire traffic does become an issue.

Okay.

> If someone can come up with a good reason why we have to do this, I would

It could well be that my reasons above don't stand up to the gains in
efficiency  that Steve's proposal might yield. But I am interested in the question

of peer symmetry versus protocol complexity, so if anyone has something
about that, I'd like to hear it too.

Regards,
Richard

---

From: [email-masked] (Greg Rabil)
Date: Wed, 8 Dec 1999 20:40:14 -0500 (EST)

> remove the details from the failover draft was made.

OK, now I'm confused.  I thought (like Richard) that we'd decided to move
the load-balancing into a separate draft because is was _not_ required for
failover.  Yes, it is nice to have, but I did not realize that we were
requiring it.  If we are, then why publish a separate draft?  Is the
failover draft now dependent upon the load-balancing draft for standard?

Greg

---

From: Steve Gonczi <[email-masked]>
Date: Wed, 8 Dec 1999 21:16:52 -0500 (EST)

>haven't done it yet, but yes, it could (make that *will*) certainly get

They are useful/needed in supporting user/vendor classing. Just a 
way to group arbitrary IP addresses together for whatever purpose.
E.g.: a group of addresses are set up to be allowed through a firewall
and you give a lease from THAT pool if the client matches 
some criteria. The above mentioned addresses form a pool.
Ted has a very nice implementation in his 3.x servers. 
You may want to take a look.

>But I am interested in the question of peer symmetry versus protocol
complexity, so if anyone has something
>about that, I'd like to hear it too.

We need to get the amount of symmetry just right ;-)

sG

---

From: Kim Kinnear <[email-masked]>
Date: Thu, 9 Dec 1999 10:20:02 -0500 (EST)

>failover draft now dependent upon the load-balancing draft for standard?

I agree with Ted.  Load balancing is still required for a
complete failover implementation as specified in the -05 draft.
As I understood it, the reason to move the details of the
algorithm out of the failover draft and into a separate draft was
because the load balancing algorithm had applicability beyond
that of failover.

For instance, it would make sense if you just wanted to balance
the load between two regular servers, without any failover
involved.  Certainly there are situations where having two
servers handling the load would be cheaper than buying the
hardware to support one big server handling the same load.
Obviously you have to split your address pools, and this is a
challenge of sorts.  Of course, today people do this by putting
some pools on one server, and some on another, and get largely
the same effect, so that load balancing I think is really
interesting in a non-failover case mostly when you have *really*
big pools with *lots* of activity -- but some cases like this do
exist.

Cheers -- Kim

---

From: Kim Kinnear <[email-masked]>
Date: Thu, 9 Dec 1999 10:26:26 -0500 (EST)

Richard said:

I don't think POOLREQ functions need to be expensive, if the
primary is implemented to know how many addresses the secondary
has and how many it's allowed (or configured) to have.  Maybe
that's going to be harder to do than I think ;-).

then Steve replied:

I am afraid it is a bit harder.  New trends in server operation
introduce complexities like arbitrary lease pools, (with
different settings and privileges), which leads to the primary
having to leaf through all of them, to hand over the prescribed
percantage of each.

Kim adds:

In my experience it depends entirely on how you implement the
accounting for lease pools.  If you keep up the number of primary
free and secondary free, then this is a very fast operation.  If
you don't, and have to calculate the numbers "on the fly", then
it isn't a very fast operation because you have to touch each
lease to determine if it is free and if so, who has it.

I would assert that you want to do it this first way in any case,
however, whether you do it on primary or secondary, because no
matter what, you don't want to have to scan all of the leases to
decide whether you need more or your partner needs more.

-----------------------------------------

Richard said:

I always assumed that POOLREQ/POOLRESP exchanges could happen any
time, and that when they did happen, there would be a period of
extra bind updates and then it would be done for a while.  But
compared to the rest of the traffic, it doesn't seem that awful,
especially since the folks who have implemented already seem to
feel that wire traffic is nothing compared to other functions
(like stable storage).

Steve replied:

It all depends on the kinds of trafic you are handling.  If you
want carrier class performance, wire traffic does become an
issue.

Kim adds:

I agree with Richard -- POOLREQ can happen an time.  From the
efficiency standpont I don't see any difference in the wire
traffic with either approach.  The actual POOLREQ is totally
insignificant in the greater scheme of things.  There is a
BNDUPD/BNDACK operation in either approach, from one server to
the other, here, so I don't find this particular argument to be
significant.

----------------------------------

Richard said:

If we have the secondary change the state of a lease to BACKUP
and update the primary, what happens in the case of a conflict?
Maybe we'd simply need another reject-reason code, but there may
be more to it than that.

Steve replied:

I would very much like to avoid cycling the backup server's expiring
leases through the primary if it can be avoided, for reasons of
efficiency, and keeping the same lease with the same server.

If someone can come up with a good reason why we have to do this,
I would very much like to hear it.

Kim adds:

I don't think there is an efficiency issue here, frankly.  As I
noted above, there is a BNDUPD/BNDACK in any case before the
backup can use the IP address for a different client.  Whether
the BNDUPD/BNDACK is initiated by the primary or secondary is the
issue, I think.

I need to think about this for a bit, since something bothers me
about the approach of lease ownership of leased leases.  The
basic design for this was done a number of years ago, actually,
and so it may take a bit to recover some of the thinking behind
the current approach.  

In addition, I'm moving my office today and we have a release
about to ship just now, so I'm going to be out of touch for at
least a few days I expect.  But I will continue to think about
this issue and try to figure out what concerns me about it.

Cheers - Kim

---

From: "David Erdmann" <[email-masked]>
Date: Thu, 9 Dec 1999 10:31:34 -0500 (EST)

My understanding was that if there was any dependency that the load
balancing was dependent on the failover draft.  But that the failover
draft could be implemented without the load balancing.

I agree that requirements for the load balancing should play an important
role in defining the failover draft, but I dont' think that we should
require load balancing in order to implement the failover.

-Dave

---

From: Steve Gonczi <[email-masked]>
Date: Thu, 9 Dec 1999 16:08:41 -0500 (EST)

>lease to determine if it is free and if so, who has it.

OK. This is certainly a good optimization. 

Let's look at the message exchange you need when you start with a 
lease in backup state, to put that lease into backup state again.

1) send a binding update, inticating the lease is active.
2) receive ack
3) send a binding update indicating the lease is now expired/released
4) reveive ack
--- at this point you can not use the lease because it is in "free" state,
not "backup".--
5) send a poolrequest
6) receive a poolreply
7) receive a binding update indicating the lease is now "backup".
8) send an ack.

I am arguing that 5)..8) is not necessary, if you put the lease back in
"backup" state
in 4).

>insignificant in the greater scheme of things. 

No argument there. It is the extra update traffic
needed to bounce back the expired leases that bothers me.

>backup can use the IP address for a different client. 

No, it can not use the address for a new client, 
because it no longer owns it!

sG

---

From: Richard Jones <[email-masked]>
Date: Thu, 9 Dec 1999 18:20:32 -0500 (EST)

> I am arguing that 5)..8) is not necessary, if you put the lease back in

Having had another day to think about this, I find another question.
One of the design goals in RFC 2131 is to 'Retain DHCP client
configuration across DHCP client reboot. A DHCP client should ...be
assigned the same configuration parameters (e.g., network address)
in response to each request.'

It seems like having to cycle leases through the primary each time
their state changes away from active makes this very difficult.Any
comments on this? Am I drifting over to Steve's point of view?
Don't know yet - the more I think...

Thanks, Steve. This gets my mind off shipping dates!

Regards,
Richard

---

From: Steve Gonczi <[email-masked]>
Date: Thu, 9 Dec 1999 19:00:50 -0500 (EST)

>Don't know yet - the more I think...

I do not think there is a problem with this, because the 2 servers
hopefully replicate all information relevant to clients.

Stay tuned, I am composing another message, where I am voicing some 
more dissenting views. ;-)

sG

---

From: Steve Gonczi <[email-masked]>
Date: Thu, 9 Dec 1999 19:05:11 -0500 (EST)

Hello,

I pondered a bit more on the POOLREQUEST & related issues.
I respectfully put forward, that the current lease status 
design is flawed. The error we have made is reminescent of
the Y2K glitch.

Leases are going through the EXACT SAME changes on both the
primary, and the secondary. (Get assigned, expire, etc).
These identical changes, and nothing else, should be reflected 
in the lease status. The lease ownership is an entirely 
different kind of thing. Hence:::

The lease ownership should be a separate attribute, 
(handled independently of the lease status).

If it was so, leases ownership could be transferred 
even if the lease was in active state, or if it was a
reservation (BTW we need a new lease state for 
reservations). 

IMHO this would make it a lot easier to conceptualize 
things:

a) Leases belong to either the primary or the secondary.
This is decided by a POOLREQUEST/REPLY negotiation.
(The Binging Update message needs a 
new option for lease ownership. )

b) The ownership assignment remains in effect, until 
a renegotiation takes place. The sole purpose of a renegotiation
is to reflect configuration changes ( e.g: new lease pools
are added, or the backup pool percentage has been changed).

c) Negotiation takes place after synchronization is complete.

d) The backup server does not start serving clients until 
the address ownership negotiation is done.

e) Leases can be given, or taken away by the primary server
at its discretion, even if they are being used by a client.
This is no big deal, as either server is allowed to renew
a lease. It also makes it easy to set the exact backup/primary
lease percentages.

I have a bit more to say, but let's see how this goes over ;-).

sG

---

From: Richard Jones <[email-masked]>
Date: Fri, 10 Dec 1999 12:47:31 -0500 (EST)

> hopefully replicate all information relevant to clients.

Hold on a sec. With our current setup, let's say the secondary is
configured via hash bucket assignment to respond to a DISCOVER
from client A. It assigns an IP, and sometime later client A releases
it. The address then gets cycled back to the primary. Some time after
this, client A does either another DISCOVER or a REQUEST, with
every right to expect the secondary to make all efforts to give it back
the same IP. However, since the address has been cycled back to
the primary, and the mechanism for getting addresses back to the
secondary is based more on whether or not the secondary is
running low on addresses, I think this design goal goes by the
wayside. Am I missing something?

I guess the only way to deal with this right now is for the secondary
to issue a POOLREQ every time it relinquishes an IP back to the
primary. I don't think this is what we had in mind with the
POOLREQ/POOLRESP messages. If I were to be real stubborn
about my earlier argument concerning the primary's priority, I'd
say this was okay, but now I'm wondering if it's necessary.

---

From: Steve Gonczi <[email-masked]>
Date: Fri, 10 Dec 1999 15:29:48 -0500 (EST)

> wayside. Am I missing something?

Your analysis is correct. One more reason to keep the released/expired 
lease with the backup server, and NOT transferring ownership.

>primary. 

We could also trigger an automatic,immediate binding update from 
the primary, setting it back to backup. 

The simplest, obvious solution is to set it back to BACKUP and 
not to FREE.

sG

---

From: Kim Kinnear <[email-masked]>
Date: Tue, 21 Dec 1999 11:07:34 -0500

Steve, et.  al.,

I've been thinking about the POOLREQUEST issues that you have
brought up.

First, my previous analysis was flawed, as you pointed out, in
that it would take two BNDUPD/BNDACK sequences for an expired
address to transit from the secondary to the primary and back
again today, and only one with some of the proposals that have
been kicked around recently.

I've spent some time trying to figure out what problem we are
trying to solve here.  There seems to be an implicit assumption
that an expiring (or released) lease on an IP address that the
secondary server leased out would "naturally" want to flow back
to the secondary -- and thus the concept of ownership of leased
IP addresses that you have proposed.

I see two cases here:

   1.  In the general case, where a lease frees up on the
   secondary that was originally leased on the secondary, I don't
   think there is a problem to solve -- at least then.  This is
   because after the lease was originally granted (and long before
   it frees up), the primary and secondary servers are going to
   have to have replenished the secondary's store of available
   leases.  The shortage of available IP addresses on the
   secondary is an event widely separated in time from the actual
   lease expiration (generally).  There is no telling a priori
   which server needs a newly available IP address more -- the
   primary or secondary, and it certainly would have no
   relationship to who originally leased it.

   2.  In the load balancing case, there would be some value in
   having an expired or released IP address end up as part of the
   available pool of the server who most recently leased it to a
   client, since there is good reason to try to give that client
   back the same IP address if it shows up asking for an IP
   address again.  The loadbalancing algorithm will ensure that it
   shows up at the same server the second time, so it makes sense
   for the same server to have it available.

Of course, if that server fails, then that client isn't going to
get the same IP address back if it asks while the other server,
the only one running, is in COMMUNICATIONS-INTERRUPTED state, but
that is ok -- at least it gets *some* IP address.

---------------------

So I do see two problems to solve (perhaps not exactly the ones
originally giving rise to your proposal):

    1.  Keeping the available pools of each server balanced in the
    way that they have been configured.  This needs activity at
    two discrete times:

	a.  Whenever either server leases out an IP address, some
	rebalancing of the available pools might need to occur.

	b.  Whenever either server senses a lease expiration or
	release, there is the question of what to do with it.

    2.  At time (b) above, trying to ensure that the client that
    used it last can get it again.

I'll make a proposal to solve these problems in a separate mail.

Cheers -- Kim

---

From: Steve Gonczi <[email-masked]>
Date: Tue, 21 Dec 1999 14:36:42 -0500

Kim says:

>There seems to be an implicit assumption
>to the secondary 

Yes. As the protocol is written today, they do. 

>   1.  In the general case,...

Do you mean the case _without_ load balancing? 
I have no issue with that at all. Works fine as-is.

>   back the same IP address ...

Absolutely. 

>	rebalancing of the available pools might need to occur.

I am not convinced this is worth doing. It reminds me of the centrally
planned
economies of the Soviet block. You would have to be pretty smart about
rebalancing , including looking ahead to see what expirations are expected
in the near future to have a hope of making things better instead of worse. 
(Mind you, there is no way to foresee how many clients are going to boot 
or release in the next minute).

Imagine this: Oops, I am low on leases. Give me some. Oops, forget that, 
a whole bunch just freed up. 

Overall, you may be doing a lot of work, to end up in a situation that is 
slightly worse than it would be without your attempts to fix it.

It may make sense to make pool percentage adjustments based on some
statistics, over longer periods of time. (Intelligent self-tuning).

>	release, there is the question of what to do with it.

Seems obvious to me. Keep it with the server who had it in the first place.
Why not?

---

From: Steve Gonczi <[email-masked]>
Date: Tue, 21 Dec 1999 15:03:26 -0500

>>Yes. As the protocol is written today, they do. (go back to the primary)
>
>         No, they flow back to the primary, then on next POOLREQ they
>         go wherever they are needed.  Or did I miss something
>         in what you said?

 I am talking about what happens during normal operation. ( Both servers
are up). Are you assuming periodic POOLREQ -s in normal mode?
My assumption: we are still debating if that is the right solution.

>         My point is that doing what you suggest does not even begin
>         to handle the issue of actually balancing the pools on the two
>         servers, so that if one goes down, then the other may not have
>         enough IP addresses to handle the outage.

Well, there is no way to foresee which one goes down. Hence the best bet 
is to split the lease pools evenly to begin with.

Sugsequently, the "randomness" of client transactions and expirations
are probably as good a way of maintaining the 50-50 as any balancing
algoritm we could come up with. 

>         Leases run for hours to days to weeks, and when they expire
>         there is no reason to think that the machine which originated
>         the lease is necessarily the one that needs more IP addresses.
         
>         Cheers -- Kim

Like I said, one server may need addresses this second, but that may change
the next second. Hence, any re-balancing you attempt, is likely 
going to make things worse. 

sG

---

From: Kim Kinnear <[email-masked]>
Date: Tue, 21 Dec 1999 15:34:47 -0500

>statistics, over longer periods of time. (Intelligent self-tuning).

          That is my point of having a threshold in the proposal that
          I sent.

          
          Cheers -- Kim