Subject: DHCP failover--using only TCP (no UDP polling)

From: "David Erdmann" <[email-masked]>
Date: Mon, 3 May 1999 11:06:36 -0400 (EDT)

Hi,

At the last failover () meeting we were discussing how to use TCP for this
protocol.  Here is a portion of the notes from that meeting.
(the full notes of the meeting in [email-masked] group 3/99)

>    We agreed to try to see if we could figure out during the rest of
>    the meeting how to properly use TCP for this protocol.
>
>    When discussing that, we determined that polling would be a problem
>    in TCP.  We considered doing just the polls in UDP.

>    We agreed to do just the polls in UDP.  Why?

>	1.  because if the TCP connection is busy, you can't necessarily
>	tell if the other server is up or not

>	2.  because if the TCP connection may not determine that the
>	other server is not up quickly enough

>    At the very end of the meeting, Ted suggested that we try to do
>    polls in TCP as well, and just accept that either up or down
>    detection was going to be spotty.  We talked about it a bit, but
>    time was short.  We did not agree to this approach, but did agree
>    that it merited further discussion.  He agreed to raise this on the
>    list for further discussion.

I propose that we could get rid of UDP completely, and just rely on TCP.
I am sure that I have missed some fine points, so if somebody could
point out any faults that would be appreciated.

Here is what I propose:

It is essentially based on the idea that we reset the partner connection
timeout counter every time we receive any data from the partner;  whether we
receive a ping, or actual data.

I think that this approach is valid because we should be getting all of the
latest
information from the other server before we start serving its addresses.
So if the connection were to go down, then we would still like to get all
of the information from the pipe before continuing.

We will need two levels of polling.  We can swap which one is responsible
for initiating which polling, it makes no difference.

---------------
A.
1. The primary's timer (period of T) goes off, sends a ping (P1) asking if
	the secondary is up.
2. Upon receipt of this ping P1, the secondary sends a reply to the primary
(P1r).
	The secondary also resets its timer for not having heard from the
	primary.
3. The primary receives the secondary's response P1r, and resets its timer.
	It will send another ping P1 when this timer (T) has run out.
	If the primary does not receive the reply P1r after a certain lets
	say (X x T)...it will drop the connection, and proceed as if the
	secondary is down.
---------------

---------------
B.
Now this level of polling is to handle the case where the primary has
buffered
a lot of updates to be sent to the secondary, and the secondary has been
processing this information for a long period of time, greater than the ping
timeout time.  If the primary puts a ping request on the queue of
information
for the secondary to process, then it will not receive it in time...this was
a
major concern when we were talking at the Failover meeting earlier this
year.

1. The secondary has another timer (T2) that is reset every time it receives
a
	ping P1 from the primary, or it sends data to the primary.  It has a
timeout
	period (Y x T), where ( 1 < Y < X ).
	It also has 2 flags, one if it has seen data (DATA) since T2 last popped,
	the other if T2 has popped, (POP).

2. If we receive information from the primary, that is not P1:
	if( POP )				//if the timer has popped
	{
		send( primary, P2r )	//send the primary the ping
		reset( T2r, DATA, POP ) //reset our timer
	} else {				//timer has not popped
		DATA = true			//set our flag to true
	}

3. If the timer (T2) pops:
	if( DATA )				//if we have received data
	{
		send( primary, P2r )	//send the primary our ping
		reset( T2r, DATA, POP )	//reset our timer
	} else {
		POP = true			// set timer flag to true
	}

4. If the primary receives a T2r (inform ping from secondary),
	it should behave as normal and reset its T timer.
--------------

I think that this addresses the problems raised at the failover meeting:

>    We agreed to do just the polls in UDP.  Why?
>	1.  because if the TCP connection is busy, you can't necessarily
>	tell if the other server is up or not

If the TCP connection is busy, that is fine, that is exactly what the plan
B above takes care of.  If it receives information it will send out an
inform
ping.

>	2.  because if the TCP connection may not determine that the
>	other server is not up quickly enough

If you want speed, then just set the timeout values differently.
I think that this would be up to the implementation to decide what priority
to process events.  I don't think that this is an issue with the above plan.

-Dave

-------------
Dave Erdmann
Nortel Networks, NetID Group
mailto:[email-masked]

---

From: "Bernard Aboba" <[email-masked]>
Date: Mon, 3 May 1999 13:31:14 -0400 (EDT)

>I propose that we could get rid of UDP completely, and just rely on TCP.

I would agree that UDP is not necessary in this protocol. BGP provides
a good example of how many of the issues can be dealt with effectively
in a TCP-based protocol.

>receive a ping, or actual data.

Yup. The BGP hold time is defined as the maximum time that may elapse
between KEEPALIVE or UPDATE messages. Note that in BGP the hold time
is negotiated to the lower of the values of the peers; the minimum
recommended hold time is 3 seconds.

>I think this approach is valid

It is indeed valid, as demonstrated by the success
of BGP.

>If the primary puts a ping request on the queue of
>information for the secondary to process, then it will
>not receive it in time...this was a major concern when
>we were talking at the Failover meeting earlier this
>year.

Since the connection can be kept open by either a
keepalive or an update message, keeping the connection
alive per se is not really an issue. However, there may be
a desire to ensure latency bounds for certain high
priority messages.

The solution to this problem is to keep the size of the
update messages small so that there will not be excessive
latency. For example, if you have a 9600 bps link,
serialization time for a 1500 octet packet is 1563 ms.
So a critical message stuck behind a 1500 octet update
could be delayed by that much, in addition to its
own serialization, transit, and queuing delays.

The solution is to have smaller update messages.
In practice if the update unit is kept to the info
involved in a single lease, then the latency should
be acceptable less unless very severe congestion is
encountered.

Note that I use 9600 bps as an example link speed
because almost all connections will be faster, and
congestion would need to be very severe to push
the effective rate below this.

>	tell if the other server is up or not

If receipt of either a keepalive or update message is
sufficient to provide information on whether the
server is up then this should not be a concern.

>	other server is not up quickly enough

I think that the issue here is congestive backoff. On congested
links, the latency will increase as TCP backs off. Since this
occurs due to dropped packets or increased RTT, there is a good
reason for it, namely that the network cannot really support
a higher rate. In such situations, the question should be asked
as to how the UDP-based failover protocol would perform under
the same circumstances. As long as the latency does not get
so high that the hold time can't be maintained, I think that
TCP will perform satisfactorily. If the latency is higher than
that, then network conditions are likely to challenge UDP
as well.

Note that DHCP failover traffic is mission critical so that it
probably makes sense to ensure that it receives appropriate
QoS (i.e. low drop preference).

---

From: Steve Gonczi <[email-masked]>
Date: Mon, 3 May 1999 13:34:41 -0400 (EDT)

Dave,

We have retained UDP in order to provide an "instant" out-of-band messaging
channel.
This is not so easy to do with TCP.

sG

-----Original Message-----
From: David Erdmann [mailto:[email-masked]]
Sent: Monday, May 03, 1999 11:12 AM
To: Multiple recipients of list
Subject: DHCP failover--using only TCP (no UDP polling)

Hi,

Here is what I propose:

---------------
A.

---------------
B.

	It also has 2 flags, one if it has seen data (DATA) since T2 last
popped,

-Dave

-------------
Dave Erdmann

---

From: "David Erdmann" <[email-masked]>
Date: Mon, 3 May 1999 14:27:24 -0400 (EDT)

Steve,

I do not immediately see the need for the out-of-band messaging.
What messages do you think we would need to pass that have to be
out of band?

The only out-of-band traffic that I can think of would be state
transition of the partner.  Under what circumstances would it be
necessary to have out of bound communication?

-Dave

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 3 May 1999 14:26:31 -0400 (EDT)

> I would agree that UDP is not necessary in this protocol.

I'm not so sure that this is a great idea. I think that the DHCP group
should be leading the effort to multicast-enable as many protocols as
possible, and this one seems like a good candidate. Multicasting would
provide a lot of functionality to a replication protocol, allowing
multiple backups to handle multiple primaries easier than a TCP-based
transaction would allow (it would of course provide a lot of
functionality to DHCP in general, something that I would like to see
more time and effort spent on discussing).

Going to TCP exclusively will preclude the use of IP multicasts.

-- 
Eric A. Hall                                            [email-masked]
+1-650-685-0557                                    http://www.ehsco.com

---

From: "David Erdmann" <[email-masked]>
Date: Mon, 3 May 1999 15:12:34 -0400 (EDT)

>  Eric Hall wrote:

Eric,

If by referring to multiple backups handling multiple primaries, you
implying that multiple backups can be desginated for a single primary,
then that is not the current aim of this failover protocol.  I thought
that we were looking for a single backup solution for this current
draft, and that having multiple backups for a single primary would 
possibly the case for future different failover protocols.

Using multicasts in a failover protocol that is trying to communicate
between 1 primary and 1 backup does not make sense to me.  I am not clear
how multicasting only some of the traffic will help.

I think that using multicasting for backing up a primary to multiple 
backups is definitly worth exploring however.

-Dave

---

From: "Eric A. Hall" <[email-masked]>
Date: Mon, 3 May 1999 15:14:00 -0400 (EDT)

> backups is definitly worth exploring however.

My concern is that by "going to TCP" we will make it more difficult to
implement multicasting later, simply because the entire process will
have to be reinvented. This only increases the likelihood that it will
not happen, IMHO.

---

From: "David Erdmann" <[email-masked]>
Date: Mon, 3 May 1999 16:05:20 -0400 (EDT)

Steve,

If we can determine that there is a need for this out of bound
traffic then I agree that we should consider UDP.  However I 
do not see why we need out of bound traffic.

As a side note I think that from a design and implementation point of 
view that we should be trying to only use one protocol unless there
are obvious and considerable advantages to using both.  Consider
the added complexity of having 2 protocols to deal with.

Lets take security as an example.  At some point we want to implement 
security over the communications between the two servers.  Security
(encryption, and authentication) is pretty hard to get right on 1
protocol let alone 2.  I don't think that we have to look too far to
see examples of programs in use on the net that have had security
problems.

I see using one less protocol as one less area for things to go wrong.

-Dave

---

From: "David Erdmann" <[email-masked]>
Date: Mon, 3 May 1999 16:04:25 -0400 (EDT)

> a desire to ensure latency bounds for certain high

I am not sure that there are any messages that need any guarantees
as to latency bounds in this failover protocol.  My understanding
of the UDP messaging was that its' only purpose was to provide
ping capabilites, while passing some additional information with the
ping.

> be acceptable less unless very severe congestion is

I agree that by making smaller messages you will reduce latency of
processing
a single message, but I don't think that it will help in this case, and I
don't think that we want to shrink our update messages any more.

We have a base amount of information to pass.  The whole problem
revolves around the assumption that we have a lot of information in
the TCP buffer.  If we go and say that we want to reduce the size
of each update, then I can equally come back and say that I want
to increase the number of updates that I want to send.  So in essence
we still have the same amount of raw data buffered to be sent out,
that is going to take a while.  Plus if the receiving server is
heavilly loaded from other connections, the bandwidth of the connection
might not be the bottleneck.

> server is up then this should not be a concern.

This is a concern because if the Primary is sending a lot of data
to the secondary, and the secondary has no information to send to
the primary, and the primary is responsible for initiating a keepalive,
then the primary will not be getting any information from the
secondary.  The primary will assume that the secondary is down,
but the connection has not thrown an exception yet.

-Dave

---

From: "David Erdmann" <[email-masked]>
Date: Mon, 3 May 1999 16:05:50 -0400 (EDT)

Eric,

That is a valid concern.

So it sounds that you are saying that you would prefer to use UDP 
for the failover protocol, no TCP at all.

It was my impression from the last failover meeting that we were going
to go with TCP and UDP.  My suggestion is to get rid of the UDP
since we were only going to be sending keep-alive information over
it along with some sync information.

On the note of expandability to multiple backups for a single primary:
I agree that it would be nice to make this protocol draft as easy as
possible to expand upon as possible.  It is not clear to me how creating
a failover protocol that is dependent on TCP and UDP would be easier
to expand upon than just TCP or UDP alone.

I am not sure that the idea of future use of multicasting was presented
during the decicion to go from all UDP to TCP/UDP in February.
But do we want to layout how we want to do n level backups before we
finish 1 level backups?  I doubt it.

-Dave

---

From: Ted Lemon <[email-masked]>
Date: Mon, 3 May 1999 18:06:16 -0400 (EDT)

> have to be reinvented. This only increases the likelihood that it will

The benefit of multicasting to the interserver protocol, even a
fully-blown one, seems dubious.   Since we are not implementing a
more-than-two-way protocol here, it seems particularly dubious.   I
see no point in considering this issue in decisions about the failover
protocol - if after having learned what we learn from the failover
protocol, we decide that it's possible to implement an N-way protocol,
the protocol is clearly going to change anyway, and we can consider
moving to UDP+Multicast then.   There is absolutely no need to be
concerned about this now, since there can only ever be two peers in
the failover protocol.

			       _MelloN_

---

From: "Bernard Aboba" <[email-masked]>
Date: Tue, 4 May 1999 02:14:27 -0400 (EDT)

>ping capabilites, while passing some additional information with the

Keepalive messages running over TCP can also be used for this purpose. 

>We have a base amount of information to pass.  

The issue is really batched updates. If you can squeeze the
keepalives in between the updates, then you can run them
over TCP without an issue. Allowing large batches of updates
will increase the jitter in keepalive arrivals and increase
the probability that you will not be able to squeeze them
in within the hold time, which could conceivably be quite
short (i.e. BGP default hold time of 3 seconds). 

>Plus if the receiving server is heavily loaded from other 
>connections, the bandwidth of the connection

I am curious as to how what traffic levels are anticipated
here. Are we talking about receiving servers that have
high CPU loading or are experiencing high failover
traffic levels? In either situation, we would probably
see an effect on UDP as well as TCP. 

>but the connection has not thrown an exception yet.

The solution to this is to send Keepalive messages in 
both directions. That way the secondary will be required
to send at least one message within the hold time.

---

From: "Bernard Aboba" <[email-masked]>
Date: Tue, 4 May 1999 08:55:19 -0400 (EDT)

>more-than-two-way protocol here, it seems particularly dubious.  

It should be noted that multicast only generates
traffic reduction to the extent that traffic passes 
over the same path. This occurs at the expense of 
increased routing system state. In general, the 
benefits of multicast do not exceed the costs until
substantial numbers of systems are involved. That 
isn't the case for this protocol, nor is it clear
that this test would be met for successors. Moreover,
given the kind of data that is being multicast, 
reliable multicast transport would probably be 
required. Solving that transport problem is a
considerable problem in itself. IMHO, multicast is
not something to worry about here.

---

From: "Bernard Aboba" <[email-masked]>
Date: Tue, 4 May 1999 09:02:01 -0400 (EDT)

>the added complexity of having 2 protocols to deal with.

In general, only a single protocol should be used unless there are
very compelling reasons to the contrary, since using two protocols
complicates deployment considerably. For example, use of both 
UDP and TCP will complicate firewall setup. It will also make 
securing the protocol more difficult.

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 4 May 1999 11:02:40 -0400 (EDT)

> over the same path.

I was thinking more about the one-to-many benefits of having a primary
send a single instance of lease data to multiple (highly distributed)
backups simultaneously. I wasn't thinking about utilization.

> required.

Maybe. I don't know.

---

From: "Eric A. Hall" <[email-masked]>
Date: Tue, 4 May 1999 13:05:40 -0400 (EDT)

> If you aren't thinking about utilization, you probably shouldn't be
> thinking about multicast.

There are plenty of examples where multicasting's one-to-many feature is
the primary focus, instead of utilization. NTP is one of these examples.
NTP uses trivial amounts of bandwidth over the long haul. The real
benefit to using multicasts with NTP is that you can replicate the time
data to a widely distributed set of hosts using a single send, instead
of having to maintain separate connections for all of the hosts. The
same benefits of one-to-many could be realized here. (Side note, these
would be benefits for DHCP in general: a client that sent a query to an
mcast address would let any server anywhere on the network respond to
the query, without having to depend on bootp forwarders.)

> Even assuming that it would be possible to multicast the bulk of the
> data, the reverse-channel acknowledgement and retry path would have
> to be unicast.

Effective multicasting depends on all systems being able to communicate
via multicast. If the protocol requires acknowledgments go to a unicast
address, then this protocol is not a good candidate for multicast. Read
that however you like.

My original position is still: making this TCP-based will only make it
more difficult to do multicasting (and one-to-many replication) later.
It would seem that the protocol is already transaction-centric, so
perhaps it doesn't matter (if you're using application ACKs, then this
protcol can't be used for multicasting anyway).

---

From: Ted Lemon <[email-masked]>
Date: Tue, 4 May 1999 13:02:49 -0400 (EDT)

> > reliable multicast transport would probably be

The current failover protocol does require reliable delivery of some
messages in order to work properly.   These messages are the ones that
are involved with the highest traffic volume - the ones where you
might conceivably benefit from using multicast.   Even assuming that
it would be possible to multicast the bulk of the data, the
reverse-channel acknowledgement and retry path would have to be
unicast.   This would be hell to implement, so there should be a
really strong reason to do it.

			       _MelloN_

---

From: "David Erdmann" <[email-masked]>
Date: Tue, 4 May 1999 13:06:03 -0400 (EDT)

> short (i.e. BGP default hold time of 3 seconds). 

How would you know how frequently to insert the messages? Since
the rate of consumption of the buffered data is variable.

> see an effect on UDP as well as TCP. 

It is conceivable that there could be a high CPU load, or lots
of traffic on a NIC, coupled with high failover traffic.

> to send at least one message within the hold time. 

This would work.  I like this.  If we don't care about the query 
response, then we could just have the primary and secondary send the 
keepalive message at a given interval.

---

From: Ted Lemon <[email-masked]>
Date: Tue, 4 May 1999 13:57:41 -0400 (EDT)

> The real benefit to using multicasts with NTP is that you can
> replicate the time data to a widely distributed set of hosts using a
> single send, instead of having to maintain separate connections for
> all of the hosts. The same benefits of one-to-many could be realized
> here.

Nope.  The servers would all have to know about each other.  In the
current protocol, both servers are fairly intimately aware of each
other's state when they're in the "communicating" state.  There's
really no way to avoid that, no matter how clever the protocol gets.
So this particular benefit is one benefit that we definitely would not
get from multicast.

> protcol can't be used for multicasting anyway).

Right.   Precisely.   There are two different levels of ACKs in the
UDP version of the failover protocol that AI/Cisco and Process
Software have implemented.   Even in the TCP protocol, there are still
application-level ACKs.

			       _MelloN_

---

From: Ted Lemon <[email-masked]>
Date: Tue, 4 May 1999 13:58:14 -0400 (EDT)

> the rate of consumption of the buffered data is variable.

In order for the protocol to work over TCP, both ends need to promise
the other end to consume data at a certain rate, and neither end can
send data fast enough that a round-trip ping couldn't happen in the
required minimum ping time.   The data doesn't necessarily have to be
processed, but it has to be taken out of the pipe so that a response
can get through.   The way this would work is that, for example, each
side might promise the other to be able to buffer as many as 10
unacknowledged messages.   Each side of course responds to pings
immediately, since these presumably are cheap to process.   So when a
side has sent ten updates without receiving any application-layer
acknowledgement, it stops sending updates until it gets an
acknowledgement.    It continues to send pings at the required
interval, and the other side continues to respond to these
immediately.

> keepalive message at a given interval.

We really need to be able to establish a bidirectional working path,
and we need to be able to have fairly tight control over the ping
packets, so I would say that keepalives don't meet this need very
well.   I think we talked about using them at the interim meeting and
decided they wouldn't work, although I don't remember any more detail
than that.

			       _MelloN_

---

From: Kim Kinnear <[email-masked]>
Date: Tue, 4 May 1999 15:09:34 -0400 (EDT)

>interval, and the other side continues to respond to these

	Ted, that's astonishingly clearly stated.  Wow.

	This is *exactly* what we do today with the UDP protocol
	in order to build in a primitive flow control so 
	that one server does not swamp the other server with updates.
	Except in the UDP case, the receiving side has to
	take whatever the transmitting side gives it.
	It works quite well.  

	But ... but .. but ... I thought that we were switching 
	to TCP in part so that we didn't have re-invent flow control.

	Another reason was so we didn't have to do retrasmissions,
	and another reason was so that we didn't have out of order
	messages. (Says the soothing voice in the background.)

	If we need to do application level flow control in order
	to use just TCP so that we can avoid having to do
	retransmissions and avoid having to worry about out of
	order messages (which is no small thing, let me tell you),
	I can live with it.   

	We also wanted to use UDP to establish some sort of 
	time synchronization for those of us who wanted to support
	servers that don't have synchronized clocks. But 
	I think we could do this with TCP as part of the
	server synchronization, though I've not worked out the details.

	If you all buy into Ted's approach for polls and application
	level flow control, I think it could work.

---

From: Kim Kinnear <[email-masked]>
Date: Tue, 4 May 1999 15:09:10 -0400 (EDT)

At 01:57 PM 5/4/99 -0400, Ted Lemon wrote:
..
>Software have implemented.   Even in the TCP protocol, there are still

	Yes!

	The protocol absolutely requires an ACK from the server which
	receives a binding update, and that ACK indicates that the
	receiving server has written the binding update information to
	its stable storage (i.e., disk or database).

---

From: Kim Kinnear <[email-masked]>
Date: Tue, 4 May 1999 15:06:51 -0400 (EDT)

Dave,

Thanks for starting this discussion.   

One first point, and then I'll respond to the 
included message.

In the -03 protocol (the current draft), each 
server's idea of whether the other server is 
"up" or not is based on receipt of replies to its 
own poll messages.  This is largely so that a 
half-duplex connection will not seem "up" to one 
server, and down to the other.  It is clear that 
a round trip has occurred, which is essentially 
what is required for the protocol to work.

The rest of my responses are inline, below:

>of traffic on a NIC, coupled with high failover traffic.

	It certainly is.  There can be very high traffic on
	the DHCP client side of the server, as well as 
	high traffic on the incoming failover connection.
	The failover connection traffic is lower priority
	than the DHCP client traffic, and so could be allowed
	to back up considerably in favor of processing the
	DHCP client traffic.
>> to send at least one message within the hold time. 

	I must be missing something (or more) -- if we do
	this, then what is the point of the keepalive message?
	Doesn't the receipt of data do just as much for you?

	If you don't care about round-trip integrity of the
	connection, then you can just use the existance of
	data in your input queue as an indication that the
	other server is up.  (But I think we *do* care about
	the round-trip integrity of the connection.)
>keepalive message at a given interval.

	One major (and I think related) issue is where we
	do the buffering.  If one were to drain the TCP
	queue with high priority, and then buffer all of 
	the data inside of the server, then the TCP connection
	could be easily used for doing the polling work.  But
	that seems to lose much of the value of the TCP
	connection, since you have to build some kind of
	flow control into the connection if you use it
	this way -- just like you do with a UDP connection.
     
	I think part of the value of the TCP 
	connection is that you can just stuff information 
	down it until it blocks on the sending side, and 
	you can process it as you get to it on the receiving 
	side, thus utilizing the natural flow control
	built into the protocol (instead of building
	our own on top of UDP).  But if we use it this
	way, then we have this problem telling whether
	the other server is alive when that other server
	is really busy and not willing to process the
	messages with any kind of priority.

	In short, I can imagine (and I have seen)
	situations where one server can get so busy
	that it could hold off failover messages for
	15 to 30 seconds or more, and process only
	one or two messages during that period.  In
	the event that happens (and that isn't any
	sort of failure condition), then it is
	not all that unlikely that the connection could
	be considered to be "down" when in fact it is
	not -- if we send polls down the TCP 
	connection like we do with UDP today.

	So, we need to devise a way that one server can
	hold off on processing another server's messages
	(for good and sufficient reasons), and yet still
	somehow indicate that it is *able* to processes
	those messages -- and that it will (in the fullness
	of time).

	Couple that with the need to bring both servers
	into communication (which yields its own little
	set of issues).

	I don't see that emitting keepalives on a regular
	schedule meets those needs.

---

From: "Bernard Aboba" <[email-masked]>
Date: Wed, 5 May 1999 12:42:59 -0400 (EDT)

>Doesn't the receipt of data do just as much for you?

Yes. Receipt of either a keepalive or update message is sufficient
to keep the connection up. The question was about a half-connection
that had not received any update messages; how would you know that
it was still alive? The proposed solution in that case would be to 
send a keepalive. 

With respect to scheduling of the keepalives so that the hold time
is maintained, several implementation approaches are possible. One
is to keep a timer that is reset whenever update messages go out 
and thus guarantees that a keepalive or update message is sent within
the hold time interval. 

>the round-trip integrity of the connection.)

As for "round trip integrity" I am curious as to how 
the definition of this differs from just having both 
half-connections up. Are you looking to derive an
RTT estimate?
 
>do the buffering... we have this problem telling whether  

This sounds like an implementation issue to me. 
If a bi-directional hold time were implemented,
the implication would be that the application would
need to take precautions to make sure that the 
hold time was maintained. A variety of scheduling
algorithms can be implemented while accomodating
this requirement. Strict priority isn't one of 
them, but "priority with at least X amount of
keepalive traffic" would do. 

>those messages -- and that it will (in the fullness

Again, this seems like an implementation issue to me. 
The sending of keepalive messages does not necessarily
need to block on emptying the receive buffer of
update messages.

---

From: "Bernard Aboba" <[email-masked]>
Date: Wed, 5 May 1999 12:45:02 -0400 (EDT)

>its stable storage (i.e., disk or database).

This is what makes the implementation of this protocol using
multicast anything but simple. You not only need to implement
ACKs, you also probably need to make them reliable. The end
result would be that a unicast UDP implementation would have
very little in common with a multicast implementation, even
though both would run over UDP. Remember, UDP is not really
a transport layer -- it's a building block. The problem is
hard enough without adding reliable multicast transport to
the list of issues.

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 5 May 1999 13:51:49 -0400 (EDT)

> ACKs, you also probably need to make them reliable.

I have given up on trying to promote making this protocol multicast
aware. It is obviously too difficult.

I have not given up on pushing for a protocol that could use multicasts
however. It would seem that replication could use multicasting quite
easily, where updates were sent to the mcast address, and each member
also sent out periodic state-markers or checksums or something,
indicating the transactions they have seen and recorded. If any members
are out-of-synch, they could initiate synchronization using unicasts
(and get the benefits of TCP then).

Actually, by having DHCP clients bind and send to a multicast address
(instead of using 0.0.0.0 and bootp forwards), all of the lease data
would be seen by all of the servers anyway, effectively providing
replication for free.

The biggest problem would come if the network suffered a meltdown and
all of the members got out of synch, which could require a lot of
synchronization work to recover from.

---

From: Kim Kinnear <[email-masked]>
Date: Wed, 5 May 1999 13:48:31 -0400 (EDT)

>hard enough without adding reliable multicast transport to

I'm not excited about making the problem harder, yet
while we're on this subject, I would observe that I
don't *think* the ACK's have to be reliable, if
I understand what we're all talking about here.

I think the sub-topic we've been discussing here is
the use of multicast for the distribution of binding
updates.  The details have been assumed.  If we dig
down a bit, then I think the primary would multicast
the binding updates to a bunch of secondaries.  I'm
guessing that the best approach would be for the
secondaries to unicast the ACK's back to the primary,
but they too could be multicast.  At some point,
if the primary didn't get an ack from every secondary
it might re-send the multicast or it might unicast
to a specific secondary.

At some point the primary/secondary distinction 
starts to disappear, and they all are multicasting
updates and ACKs to everyone.  I don't think any
of this needs to be reliable, if one is willing
to build the necessary retry logic into the
server.  Having built pretty much all of this
in a primary-secondary scenario out of UDP,
I can say that it isn't that difficult to do.

So, I conclude (waving my hands) that reliable 
multicast isn't necessary.  FWIW.

Cheers -- Kim

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 5 May 1999 14:58:19 -0400 (EDT)

> updates and ACKs to everyone.

Yes!

All servers would essentially be primaries, and the difference between a
primary vs a secondary would only depend on how it was configured for a
particular per-subnet. There would be no need for a difference in the
actual traffic.

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 5 May 1999 15:02:35 -0400 (EDT)

>         In addition to changing the broadcast behavior of
>         clients, you would also have to change the renewal
>         behavior since renews are unicast to the DHCP server
>         and ACK's are unicast back to the DHCP client by the
>         server.

Yeah. Without thinking about this too hard, it would seem that the
entire lease negotiation process (including the parts that are unicast
today) would have to be done via multicast in order for other servers to
be "witnesses" to the complete process.

There are other challenges here. Eliminating bootp forwarders obviously
has merit (I hate them), but they also provide data that is useful in
the lease negotiation process today (subnet selection, for example).
Maintaining backwards compatibility with bootp is an issue too. These
are all resolvable though.

---

From: Kim Kinnear <[email-masked]>
Date: Wed, 5 May 1999 14:59:43 -0400 (EDT)

..
>would be seen by all of the servers anyway, effectively providing

	An interesting approach.

---

From: "Bernard Aboba" <[email-masked]>
Date: Thu, 6 May 1999 04:07:53 -0400 (EDT)

>I think the primary would multicast
>secondaries to unicast the ACKs back to the primary,

The primary can certainly keep a membership list,
track the ACKs and retransmit as required (most
likely via unicast). However, some questions will arise: 

1. Retransmit timers. To make sure that you weren't
retransmitting too often, you'd probably need to
compute an RTT estimate for the members.  

2. Window advancement. A policy would be needed
on when the next update could be sent in the absence
of ACKs from all members. So you'd need to think
about when the window was advanced, window sizing,
etc.

3. Membership. This would include the policy for
how a secondary joins the membership group, and
under what conditions it is to be removed
(number of retransmissions and/or time without an ACK). 

There is considerable research and knowledge
that can be utilized in answering these
questions so that the problem is by no means
insolvable. However, when you're done answering
the questions above, I'd wonder how much code 
you'd still have in common with
the previous UDP implementation. In reliable
multicast, the devil is in the details, and
protocol often start off simple, only to gain
in complexity as corner conditions are encountered.

>multicast isn't necessary.  FWIW.

Actually, what you've just described is a variant
of reliable multicast transport (albeit with 
unicast ACKing and retransmission).