Hot Downloads

Resolving Cisco Switch & Router ‘DHCP Server Pool Exhausted-Empty’ Error – Client IP Assignment Failure

Written by Administrator. Posted in Cisco Switches - Catalyst / Nexus Switch Configuration

4.55555555556 1 1 1 1 1 Rating 4.56 (9 Votes)
Resolving Cisco Switch & Router ‘DHCP Server Pool Exhausted-Empty’ Error – Client IP Assignment Failure - 4.6 out of 5 based on 9 votes

cisco-switch-router-dhcp-server-conflicts-1In previous articles, we showed how it is possible to configure a Cisco router or Catalyst switch to provide DHCP server services to network clients. Everything usually works without a problem, however there are times when the Cisco DHCP server stops assigning IP addresses and we need to look into the issue and resolve it as quickly as possible. System messages such as ‘POOL EXHAUSTED’, ‘ASSIGNMENT FAILURE’ & ‘address pool Guest-VLAN is empty’ provide some basic information, however further investigation is required to identify the real cause.

Small-sized networks usually have DHCP services configured on their Cisco router, while large-sized networks (with multiple VLANs) assign DHCP services to their backbone layer-3 switch (Catalyst 6500, 4500, 3750 etc). The good news is that configuration and debugging commands are identical for both Cisco Catalyst switches and Cisco routers.

 

Debugging DHCP Server on Cisco Catalyst Switch & Cisco Router

The first symptoms of DHCP server issues are users nagging that they cannot connect to the network because they haven’t got an IP address, and that’s where the fun begins.

Assuming no configuration changes have been made to the Cisco DHCP server, the best way to troubleshoot the problem is to enable debugging on the dhcp server.  The debug ip dhcp events & debug ip dhcp server packets are useful debugging commands that will help us identify what is happening:

4507R+E# debug ip dhcp server packets
4507R+E# debug ip dhcp server events
Nov  6 13:46:26.742: DHCPD: Sending notification of DISCOVER:
Nov  6 13:46:26.742:   DHCPD: htype 1 chaddr 34bb.1f9b.17f9
Nov  6 13:46:26.742:   DHCPD: giaddr = 192.168.7.10
Nov  6 13:46:26.742:   DHCPD: interface = Vlan7
Nov  6 13:46:26.742:   DHCPD: class id 426c61636b4265727279
Nov  6 13:46:26.742:   DHCPD: out_vlan_id 0
Nov  6 13:46:26.742: DHCPD: Sending notification of DISCOVER:
Nov  6 13:46:26.742:   DHCPD: htype 1 chaddr 34bb.1f9b.17f9
Nov  6 13:46:26.742:   DHCPD: giaddr = 192.168.7.10
Nov  6 13:46:26.742:   DHCPD: interface = Vlan7
Nov  6 13:46:26.742:   DHCPD: class id 426c61636b4265727279
Nov  6 13:46:26.742:   DHCPD: out_vlan_id 0
Nov  6 13:46:26.742: DHCPD: subnet [192.168.7.1,192.168.7.254] in address pool Guest-WiFi-VLAN is empty.
Nov  6 13:46:26.742: DHCPD: Sending notification of ASSIGNMENT FAILURE:
Nov  6 13:46:26.742:   DHCPD: htype 1 chaddr 34bb.1f9b.17f9
Nov  6 13:46:26.742:   DHCPD: remote id 020a0000c0a8070107000000
Nov  6 13:46:26.742:   DHCPD: giaddr = 192.168.7.10
Nov  6 13:46:26.742:   DHCPD: interface = Vlan7
Nov  6 13:46:26.742:   DHCPD: class id 426c61636b4265727279
Nov  6 13:46:26.742:   DHCPD: out_vlan_id 0
Nov  6 13:46:26.742: DHCPD: Sending notification of ASSIGNMENT_FAILURE:
Nov  6 13:46:26.742:  DHCPD: due to: POOL EXHAUSTED

The key information provided by our debugging is highlighted in bold. This information tells us that our address pool named Guest-WiFi-VLAN is the DHCP Pool where we have a problem because the pool is empty, which means the DHCP server has no more free IP addresses to assign to new clients.

The next step is to understand why there are no more free IP addresses. A common reason is that there are more clients in the specific VLAN requesting IP addresses, than what the DHCP server can assign. Let’s see if this is the case.

First, we take a look at the configured IP address range for that VLAN/Pool. Note that our Guest VLAN is assigned to VLAN7:

4507R+E# show run
...output omitted....
!
ip dhcp excluded-address 192.168.7.1 192.168.7.20
!
ip dhcp pool Guest-WiFi-VLAN
 network 192.168.7.0 255.255.255.0
 default-router 192.168.7.1
 dns-server 8.8.8.8 8.8.4.4
 lease 0 2
!
…output omitted
Looking at our DHCP server configuration, we’ve reserved the first 20 IP addresses from the Class C network 192.168.7.0, which leaves us with 234 available IP addresses.  At the same time, our DHCP server is configured to provide a 2 hour lease (lease 0 2) for each IP address. This means that every 2 hours, the DHCP lease is automatically renewed between the DHCP server and client – assuming the client is still connected to the network. If the client is disconnected from the network when the renewal time arrives, the IP address assigned is then released by the DHCP server, moved back into the VLAN’s DHCP pool and made available for assignment to another client.

Let’s check and see how many clients have been assigned an IP address for VLAN7:

4507R+E# show ip dhcp binding | inc Vlan7
IP address      Client-ID/Hardware address   Lease expiration        Type       State      Interface
192.168.7.81    019c.65b0.3760.e3       Nov 06 2014 05:19 PM    Automatic  Active     Vlan7
192.168.7.92    012c.2997.58a3.b5       Nov 06 2014 05:42 PM    Automatic  Active     Vlan7
192.168.7.134    0114.8fc6.bd62.f2       Nov 06 2014 04:41 PM    Automatic  Active     Vlan7
 We should begin by explaining that the pipe used in the command show ip dhcp binding | inc Vlan7 helps us filter the output that will be provided, so that we only see information that includes the word Vlan7. If we did not include the | inc Vlan7 filter, the command line would return DHCP information for other Vlans – assuming the switch was configured as a DHCP server for them.

The output surprisingly shows us that we have only 3 clients to which IP addresses have been allocated. So the question now is where did all the rest of the 231 (234-3) IP addresses go?

Another useful command to check the DHCP pool usage is the show ip dhcp pool. It provides the overall usage of the pool alongside with the total addresses, leased and excluded addresses:

 4507R+E# show ip dhcp pool
Pool Guest-WiFi-VLAN :
 Utilization mark (high/low)   : 100 / 0
 Subnet size (first/next)        : 0 / 0
 Total addresses                  : 254
 Leased addresses               : 3
 Excluded addresses            : 251
 Pending event                    : none
 1 subnet is currently in the pool :
 Current index               IP address range                    Leased / Excluded / Total
     0.0.0.0               192.168.7.1 - 192.168.7.254               3     / 231    / 254  

Here we can again confirm that from the 254 total IP addresses, 251 are excluded and 3 are leased. Note that the Excluded addresses includes the manually excluded and conflicted IP addresses.

The Current index column shows the next IP address that will be assigned by the DHCP server. Under normal operation, we would expect to see and IP address within the 192.168.7.0 network, however the value of 0.0.0.0 means that there are no more available IP addresses to lease.


The next step is to check the DHCP server for possible conflicts using the show ip dhcp conflict command – we are sure to find something here:

4507R+E# show ip dhcp conflict
IP address          Detection method   Detection time          VRF
192.168.7.59      Ping                         Sep 10 2014 06:17 AM                                    
192.168.7.62      Ping                         Sep 10 2014 06:35 AM                                    
192.168.7.21      Gratuitous ARP         Sep 10 2014 09:58 AM                                    
192.168.7.67      Gratuitous ARP         Sep 10 2014 10:54 AM                                    
192.168.7.69      Gratuitous ARP         Sep 10 2014 12:08 PM                                    
192.168.7.96      Gratuitous ARP         Sep 10 2014 12:11 PM                                    
192.168.7.100     Ping                        Sep 10 2014 01:37 PM                                    
192.168.7.129     Gratuitous ARP        Sep 11 2014 02:13 AM                                    
192.168.7.156     Gratuitous ARP        Sep 11 2014 02:19 AM                                    
192.168.7.164     Gratuitous ARP        Sep 11 2014 04:52 AM                                    
192.168.7.158     Gratuitous ARP        Sep 11 2014 05:46 AM                                    
192.168.7.230     Gratuitous ARP        Sep 11 2014 04:35 PM                                    
192.168.7.236     Gratuitous ARP        Sep 11 2014 09:00 PM   
…output omitted

To save space, we had to remove the rest of the command’s output. Surprisingly enough, we found that all 231 IP addresses were listed in the dhcp conflict table.  As shown above, the IP addresses are listed by date of conflict with the older entries shown first.

Understanding the DHCP Conflict Table & Cisco DHCP Server Functionality

Before a Cisco DHCP server hands out an IP address to a client, it always ARPs and then pings the address to be handed out to make sure no one is using it.

When a Cisco DHCP server discovers a conflict, it will place the IP address into the conflict table stating the address was conflicting and how it came to that conclusion, as noted under the Detection method column.

If for any reason the client who is already using the IP address that is about to be handed out by the Cisco DHCP server, does not respond to the ping from the DHCP server, the DHCP server will lease out the IP address since it cannot identify any conflict issues.

The first thing the client will do once the IP address assignment is complete, is to send out a gratuitous ARP message with its new IP address. If not reply is received, then it is safe to assume no one else is using it. However if it does receive a gratuitous ARP reply, then it will indicate that another device on the network is already using that address.

Assuming a gratuitous ARP reply is received, the client will send a DECLINE message to the DHCP server, rejecting the IP address it was just assigned. Since Cisco DHCP server has seen two gratuitous ARP messages and discovered there is a conflict, it will move the IP address into its conflict table and assign the next available IP address to the client.

Clearing the IP DHCP Conflict Table

When the DHCP server detects there is a conflict of an IP address before or right after it is assigned to a client, it will automatically remove the IP address from the DHCP pool and move it to the DHCP conflict table.  The IP address in question will remain there until an administrator sees and clears the DHCP conflict table.

If DHCP conflicts are occurring frequently, it is only a matter of time until all available IP addresses are moved to the DHCP conflict table and the DHCP Pool is left empty with no IP addresses to hand out.

We can clear the DHCP conflict table by using the clear ip dhcp conflict * command. This will instruct the DHCP server to clear the conflict table and return all IP addresses to the DHCP Pool.  In case we have multiple VLANs and Pools, the command will affect them as well:

4507R+E# clear ip dhcp conflict *
4507R+E# show ip dhcp conflict
                 IP address        Detection method   Detection time          VRF

4507R+E#

Issuing the show ip dhcp conflict command confirms that there are no more IP addresses in the table.

The show ip dhcp pool command will now show all previously conflicted IP addresses, available to be handed out to our clients.

or                   

Articles To Read Next:

CCENT/CCNA

Cisco Routers

  • SSL WebVPN
  • Securing Routers
  • Policy Based Routing
  • Router on-a-Stick

VPN Security

  • Understand DMVPN
  • GRE/IPSec Configuration
  • Site-to-Site IPSec VPN
  • IPSec Modes

Cisco Help

  • VPN Client Windows 8
  • VPN Client Windows 7
  • CCP Display Problem
  • Cisco Support App.

Windows 2012

  • New Features
  • Licensing
  • Hyper-V / VDI
  • Install Hyper-V

Linux

  • File Permissions
  • Webmin
  • Groups - Users
  • Samba Setup