Subject: Newbie Post - Assembly Line DHCP Problem

From: RM <[email-masked]>
Date: Wed, 9 May 2001 12:05:40 -0500 (CDT)

I should probably lurk for a bit before posting, but the situation has
become critical and we need a fix.  I hope someone here can help shed some
light on this issue.

I am trying to help a manufacturing company solve a DHCP problem on their
assembly line.  They have a large number of new PC's that connect to the
network using PXE implementation.  They then begin a software image
download process.

Our problem lies in the grace period that NT and 2000 server imposes on
expired DHCP leases.  The expiration time has been set to five minutes
(no, that's not a typo), which works exactly as planned.  The grace
period, however, prevents these expired leases from being immediately
returned to the pool of available addresses.  With NT4, it's about 24
hours (per Q153072).  The DatabaseCleanupInterval registry key can be set
to any number of minutes, but this seems to have little or no effect on
the grace period for expired leases.

With Win2000, it's a little bit different.  Q261964 documents the grace
period as being four hours - with a twist.  Quote: "any lease that falls
within the grace period (inactive) may be issued by the DHCP server if the
available leases fall to 0."

Well, it DOES work as described.  Sort of.  When the scope fills, the next
database cleanup will indeed delete the "expired but within grace period"
leases so that the addresses are once again available.  Trouble is, the
database cleanup only runs once per minute (the default is 1440 minutes
but we tweaked the aforementioned registry key).  This means that once a
scope fills, there will be a delay of up to 60 seconds before the cleanup
occurs and addresses are once again available.  In our high-speed
manufacturing environment, this is not tolerable.

Does anyone know of a way to totally eliminate the grace period with these
DHCP server products?

I see several possibilities here:

1. Live with it.

2. Redesign the network to use much larger subnets.  Right now, the
software download areas are in /24 subnets.  The scopes are 200 addresses
each.  We could go to /16 subnets and then potentially create huge scopes
that would never run out of addresses - grace period would be a non-issue.

3. Switch to a 3rd party DHCP product with configurable grace periods.
What product could we use?  I *might* be able to get a Sun workstation or
even a Linux/Intel box in here for DHCP services, but I'm not going to
hold my breath...

4. Get the clients to somehow do a DHCP release before they get boxed up
and sent out the door.  I don't know if these servers impose a grace
period on released leases (or only on expired leases), so this may not be
an option.

Thoughts?

RM

---

From: [email-masked]
Date: Wed, 9 May 2001 17:54:36 -0400

From: RM <[email-masked]>

I'm not familiar with the Windows implementations, but your situation
has a philosophical difference with the situations that DHCP is
designed for: It is expected that DHCP will be used to give each
machine on the network an address, which it will use as long as it is
functioning.  The result is that one always assigns more addresses to
the pool than one has machines and exhausting the pool is not a common
problem.

I gather from your message that your machines only keep a lease while
they're doing the download, and you have far more machines than
addresses.

One way to avoid this would be to use a large, private address space
for your machines.  E.g., the reserved class A network, 10.0.0.0/8.

I expect that minimizing grace periods will not help a great deal.
I've seen DHCP response times of a minute or more in large networks,
and it can be the slowest component of network initialization.  You've
made it clear that you need to start a download faster than that.  It
would be easier to get fast network services if all the machines
obtained and kept DHCP leases at all times, so they would never have
to wait.

Dale
-- 
Dale R. WORLEY    Project Manager - UPnP        <[email-masked]>
Virata Corp.	  Applications Infrastructure   http://emweb.com

---

From: RM <[email-masked]>
Date: Thu, 10 May 2001 10:50:10 -0500 (CDT)

On Wed, 9 May 2001 [email-masked] wrote:

>has a philosophical difference with the situations that DHCP is designed
>for: It is expected that DHCP will be used to give each machine on the
>network an address, which it will use as long as it is functioning.

Very true.  In our case, once PXE is done, IP connectivity is no longer
needed until the machine reboots.  PXE is used again on the second reboot,
but not on subsequent reboots.  It would be nice if we could somehow issue
a DHCPRELEASE message once the TFTP transfer of the boot image is
complete.

>The result is that one always assigns more addresses to the pool than
>one has machines and exhausting the pool is not a common problem.

Very true again.  Our scope has 201 addresses and we never exceed 201
clients at any one time - the max is 125 or so.  The trouble happens when
we bring a new batch of 125 machines in for downloading 30-60 minutes
later (and take the old one out).  All but 75 leases are now in their
grace period.  That leaves about 50 machines that cannot connect until
Win2k's DHCP runs a database cleanup, notices that the scope is full, and
prematurely deletes all the grace period leases.  As I mentioned, this
database cleanup can be triggered every 60 seconds, so these 50 machines
are in limbo for up to 60 seconds.

>they're doing the download, and you have far more machines than

See above.  We have fewer.  Sorry I was unclear.

>One way to avoid this would be to use a large, private address space for
>your machines.  E.g., the reserved class A network, 10.0.0.0/8.

You are the fourth person who has suggested this and it is a very good
idea.  Trouble is, getting network changes made here would be like mating
elephants:

1. It's accomplished at a very high level.
2. There's a lot of kicking and screaming involved.
3. It takes 18-22 months to produce results.

MegaCorps are not nimble.

>I expect that minimizing grace periods will not help a great deal. I've
>seen DHCP response times of a minute or more in large networks, and it
>can be the slowest component of network initialization.

Our DHCP here is fast.  PXE consistently gets an address within two
seconds, provided our CRAPPY PXE server product doesn't have the four Xeon
700's pegged out at 100% utilization (this is a post for a different
list).

Thanks for your reply! (and to everyone else).

RM

---

From: [email-masked]
Date: Thu, 10 May 2001 13:01:44 -0400

Very true again.  Our scope has 201 addresses and we never exceed 201
   clients at any one time - the max is 125 or so.  The trouble happens when
   we bring a new batch of 125 machines in for downloading 30-60 minutes
   later (and take the old one out).

So: the Windows DHCP server has a 1 day grace period, you have batches
of 125 machines, and you swap batches every 30-60 minutes, so you are
stuck with an effective population of 1000 to 6000 machines.  Ow.

Might there be some way to trigger DHCP database cleanup when there
are somewhat more than 0 addresses available?  (Or, ugh, reboot the
DHCP server, if that is sufficient to get it to forget...)  And of
course, there's no way to manuall trigger it.

Dale

---

From: Ted Lemon <[email-masked]>
Date: Thu, 10 May 2001 10:23:30 -0700

The ISC DHCP server does roughly what you want - rather than having a
grace period, it simply sorts leases by expiry time, allocating the
least-recently-expired lease first.  I have never heard of this grace
period feature - I suspect, actually, that most DHCP servers you could
buy do not have it.

			       _MelloN_

---

From: RM <[email-masked]>
Date: Thu, 10 May 2001 13:34:43 -0500 (CDT)

On Thu, 10 May 2001 [email-masked] wrote:

>>clients at any one time - the max is 125 or so.  The trouble happens
>>when we bring a new batch of 125 machines in for downloading 30-60
>>minutes later (and take the old one out).

>stuck with an effective population of 1000 to 6000 machines.  Ow.

Except that this facility is running way below capacity at the moment.
They'd *love* to build that many machines in a day.  Hell, I might be a
permanent employee with benefits if that were going on.  Times are lean,
however.  Soft-landing, dot-com crash, yattayatta.

>Might there be some way to trigger DHCP database cleanup when there are
>somewhat more than 0 addresses available?  (Or, ugh, reboot the DHCP
>server, if that is sufficient to get it to forget...)  And of course,
>there's no way to manuall trigger it.

With NT4, no.  It'll sit there with a full scope as long as it takes for
the leases to exit the grace period.  Win2k will clean 'em out once the
scope fills, but the scope has to fill first.  A manual delete must happen
by hand.  My next step is to see if the obscure new Netsh thingy in Win2k
can work with DHCP leases.  If so, I might be able to script something and
trigger it with the scheduler service.  Wishful thinking, but I could be
on to something.

RM