Shot myself in the foot today.

Actually I pulled the trigger yesterday, but didn't feel it till this morning.

So we have this "new" IP Address Management (IPAM) software (InfoBlox), which also does DHCP and DNS. Well yesterday, around 11:30am, I was in the IPAM section creating a new network.  I mistyped the Network address, and had to delete it out of IPAM. I must have highlighted the user’s network which checked its check box without realizing it.  Because this morning I received a bunch of calls that users at one site could not login this morning. I know DHCP was the issue; because the user’s IPs were 169.254.x.x/16.  I jumped on the switch and used the “sh ip dhcp snooping binding” to see if the any client had received addresses.

There were a few, but their lease times were old, we set our lease time to 1 day (86400 sec). This led me to check the DHCP server, where I did a search for the Network and found it missing! In this new software the IPAM and DHCP databases are connected, deleting the Network deletes the DHCP scope for that network. Of course the reason we didn’t get any call yesterday is because all the clients had already received the leases for the day and were go to good till this morning when they tried to renew their IP addresses. I rebuilt the Network and the DHCP scope, and the clients started receiving their valid address.

The Total disruption for 10 users was about 30 minutes.      

Lessons learned: Slow down with newer/unfamiliar software.