Subject: [dhcwg] AD review of draft-ietf-dhc-dhcpv6-load-balancing

From: Ted Lemon <[email-masked]>
Date: Wed, 17 Dec 2014 15:03:28 -0500

This document is not ready for publication as a proposed standard.   The main problem is that it's woefully underspecified, because RFC 3074 was woefully underspecified, and so it gets some key points wrong.

In section 3.2, the text appears to be saying that in a DHCPv6 failover setup, a RENEW with a server identifier should be ignored if the load balancing algorithm doesn't identify the client as belonging to that server.   This is problematic, for two reasons.   First, it leads to the client attempting to renew multiple times, resulting in an increased load on the server.   Second, a REBIND would presumably be answered by both servers, so it wouldn't necessarily correct the problem.

But this leads to the reason I think this is not ready for publication: RFC 3074 never actually specifies under what circumstances load balancing is done: it leaves it up to the implementation.   It kind of implies that load balancing should be done on all requests.   But in fact it only ever mentions DHCPDISCOVER.   And indeed, I just looked at the ISC implementation, and it only does load balancing on DHCPDISCOVER.

Section 2 of this document suggests that load balancing is done for _all_ client-sourced messages, but that would mean that a REBIND message wouldn't get a response from the non-balancing server, which in turn would mean that the client's lease would have to expire before it could rebind to the correct server.

There are a couple of ways this could be addressed.   One way would be to just take out section 3.2.  Another would be to say in section 3.2 that if the server identifier doesn't match the recipient's server identifier, but the load balancing algorithm says that the server should answer for this client, and the server can answer, it should answer, and leave the rest of the text the same.   Of course, there's no way to be sure the _client_ will do the right thing with such a response.

Do we have any data about operational problems in this scenario with DHCPv4?

Anyway, this needs to be resolved before I can last-call the document.   Sorry for not catching this in previous reviews.

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Wed, 17 Dec 2014 22:18:45 +0000

Ted:

While I'd "vote" to remove Section 3.2 ... (personally) I didn't like this concept. If I recall, Andre (perhaps others) wanted this to move the client's interaction to the "correct" server. But I always felt that having the client retransmit Renews and having the elements (relay, servers) have to process these far outweighed any benefit from it -- obviously this happens when that server is down, but then only one server is dropping the packets.

The server I work on does not do this for v4 (or v6) - for packets where the Server-ID is included (and mandated by RFC 3315), load balancing doesn't apply and the requested server answers - period -- this is as described in 3.1 (without the exception).

But I think your statement "Second, a REBIND would presumably be answered by both servers, so it wouldn't necessarily correct the problem." is not correct. See section 3.1 - a REBIND does not contain a Server ID option so the server that is supposed to respond to this client's hash bucket is the only that responds. So, both server would NOT answer this request. (This does assume that the servers know when each other is operational - hence why this load-balancing feature is tied to failover -- and it says as much "Since the servers were exchanging lease information,". When the failover communication is interrupted, both servers would indeed be likely to respond as load balancing would not be used.)

Also, at least in a failover situation, even if all clients were serviced by only one of the servers (say they all obtained their lease when only one of the servers was running and are happily renewing their leases), what's the harm? Failover doesn't mean you have twice the throughput - since you must have proper server capacity to handle all of the clients from a single server (the partner server may be down for an extended period). And, eventually clients will SOLICIT and then move to the proper server.

If section 3.2 is dropped, I also think we should fix 3.1 to change the SHOULD into a MUST, and drop ", with the exception of RENEWs." Perhaps the MUST should be used anyway?

I'd just like to also confirm:

>The main problem is that it's woefully underspecified

Is there more than what followed (Section 2 & 3.2) that needs to be "specified"?

And:

> Section 2 of this document suggests that load balancing is done for _all_ client-sourced messages

   2.  Background and External Requirements

      The requirements for DHCPv6 are substantially the same as for DHCPv4,
      replacing DHCPDISCOVER with SOLICIT, DHCPREQUEST with REQUEST,
      CONFIRM, RENEW, or REBIND (as appropriate), etc.

Did you have any suggestion for how to deal with this? It was I believe intended to indicate that DHCPREQUEST in v4 could mean multiple things. Note that this is addressing the requirements, not the operation (though it is interesting that this section in 3074 doesn't mention DHCPREQUEST). Perhaps this can just be: "The requirements for DHCPv6 are substantially the same as for DHCPv4."?

Note that when 3074 was done, we were thinking there were two models for load balancing:
- One is that you had a server (or pair of failover partners) that answered DHCP requests for a subset of the client pool. In this model, there was no desire to have a server (or failover partners) handle clients from another hash bucket. Ignoring failover, suppose you had 4 servers - each would take 1/4 the clients. You could replace those 4 servers with 4 failover -pairs (each failover pair handling its 1/4 of the clients -- note that you could do load balancing between the 1/4 clients (each servicing 1/8) when using failover). 
- The other is that you had one failover pair servicing your clients and you wanted to load balancing between these two failover servers (when they were both running and in communication).

I think the first model is probably of much less interest these days (though it may be interesting for a scalable cloud solution where the hash bucket is managed dynamically to distribute the load to various "request" processors.)

Thanks for your review!

- Bernie

-----Original Message-----
From: dhcwg [mailto:[email-masked]] On Behalf Of Ted Lemon
Sent: Wednesday, December 17, 2014 3:03 PM
To: dhcwg
Subject: [dhcwg] AD review of draft-ietf-dhc-dhcpv6-load-balancing

_______________________________________________
dhcwg mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dhcwg

---

From: Ted Lemon <[email-masked]>
Date: Wed, 17 Dec 2014 21:48:50 -0500

On Dec 17, 2014, at 5:18 PM, Bernie Volz (volz) <[email-masked]> wrote:
> See section 3.1 - a REBIND does not contain a Server ID option so the server that is supposed to respond to this client's hash bucket is the only that responds.

Right, but that behavior is incorrect.   If a client is sending REBINDs, it might be because the server it's bound to is refusing to answer due to load balancing.  But it also might also be because it's bound to the correct server, and that other server is no longer reachable by the client.   In that case, the client will fail to renew its lease, because the backup server won't answer.   Not only that, but in this case the client would simply fail to get a lease until the problem with the unreachable server is corrected.

Of course, Delayed Service is supposed to take care of that, but if Delayed Service is configured, then presumably the server that is refusing service under 3.2 will allow service once the Delayed Service interval has elapsed.   And so the whole thing will have been for naught.

> Also, at least in a failover situation, even if all clients were serviced by only one of the servers (say they all obtained their lease when only one of the servers was running and are happily rene…

Right.

> Is there more than what followed (Section 2 & 3.2) that needs to be "specified"?

Well, it's pretty clear that the logic of doing load balancing on non-solicit messages hasn't been explored in any depth... :)

> Did you have any suggestion for how to deal with this? It was I believe intended to indicate that DHCPREQUEST in v4 could mean multiple things. Note that this is addressing the requirements, not th…

Only do load balancing on solicits.   The problem isn't actually in this document; it's that this document relies on 3074, which is underspecified.   So this document needs to correct for that.

---

From: "Bernie Volz (volz)" <[email-masked]>
Date: Thu, 18 Dec 2014 03:31:58 +0000

>Only do load balancing on solicits.   The problem isn't actually in this document; it's that this document relies on 3074, which is underspecified.   So this document needs to correct for that.

There's also Information-Request.

And, there's Confirm that is not sent to a specific server and having both respond is probably unnecessary (it may not do any harm since they should give the same answer).

I also think that Rebinds are fine to do load balancing on - at least when failover is present (i.e. that the partner is alive). A client that arrives at Rebind might have done so because the failover partner that was to service it just came back up (i.e., since the last Renew retransmission). Yes, there are isolated cases where this ends up in not given the client service, but in that case other things are broken (such as Relays are not forwarding the packets to both servers and best to learn that earlier).

So, we're really back to all the messages that don't specify a server-id option?

>Well, it's pretty clear that the logic of doing load balancing on non-solicit messages hasn't been explored in any depth... :)

Hum ... we've being doing that for DHCPv4 for many years with it deployed in large SP networks and also are doing it for DHCPv6 - we do more than just DHCPDISCOVER and Solicit. Things have been working quiet well (these are with failover and we use information about the failover state when deciding whether to process the received packet or not).

We do not do Renews and don't intend to.

Looking back at the draft, I do see that we are conflating the two different architectures I had outlined earlier and that is probably where things go bad as the rules are a bit different.

In the case were you want a server (or failover partners) to process only a subset of the clients, the rules can be very simple - discard packets are not in the server's hash bucket assignments. This can be done regardless of the packet type and whether the server-id option is present. Though it also means you only need to apply the logic to packets where the server-id option is not present (since the server-id option will suffice in dropping all other packets not for the server).

In the case of failover load balancing (i.e., where you want each server to handle "half" of the clients), the rules are different depending on the failover state. In normal failover communication state, load balancing applies; when not in normal communication, load balancing must be disabled.

And, if the goal is for use with failover, we should wait until failover is available (or further along)?

And, if the goal is general load balancing (not tied to failover), then anything to do with failover (or any redundancy) should be removed.

In the case where there isn't any common database (whether failover or not), you really have to have the client stick with the server that is to service it - there is no redundancy goal. So whether you do the hashing on some or all client packets doesn't really matter. In principle though, it seems that doing those where there is no server id option is simpler and better as it allows a client that previously communicated with a server (regardless of load balancing) to keep working what that server (if the server is still present). But when that client gets to a packet which is not directed at a specific server, then doing the load balancing is appropriate (Solicit, Information-Request, Rebind, Confirm).

If there is a common backing store (whether failover or otherwise), clients can be serviced by a different server when that server is known not to be response - those load balancing should only be done when the partner is known to be up and fully responsive. But it turns out the same packets are load balanced (Solicit, Information-Request, Rebind, Confirm).

- Bernie

-----Original Message-----
From: Ted Lemon [mailto:[email-masked]] 
Sent: Wednesday, December 17, 2014 9:49 PM
To: Bernie Volz (volz)
Cc: dhcwg
Subject: Re: [dhcwg] AD review of draft-ietf-dhc-dhcpv6-load-balancing

Right.