Shot myself in the foot today.

Originally shared Apr. 4 2014. Updated for context and reflection.

Actually I pulled the trigger yesterday, but didn't feel it till this morning.

So we have this "new" IP Address Management (IPAM) software (InfoBlox), which also does DHCP and DNS. Well yesterday, around 11:30am, I was in the IPAM section creating a new network.  I mistyped the Network address, and had to delete it out of IPAM. I must have highlighted the user’s network which checked its check box without realizing it.  Because this morning I received a bunch of calls that users at one site could not login this morning. I know DHCP was the issue; because the user’s IPs were 169.254.x.x/16.  I jumped on the switch and used the “sh ip dhcp snooping binding” to see if the any client had received addresses.

There were a few, but their lease times were old, we set our lease time to 1 day (86400 sec). This led me to check the DHCP server, where I did a search for the Network and found it missing! In this new software the IPAM and DHCP databases are connected, deleting the Network deletes the DHCP scope for that network. Of course the reason we didn’t get any call yesterday is because all the clients had already received the leases for the day and were go to good till this morning when they tried to renew their IP addresses. I rebuilt the Network and the DHCP scope, and the clients started receiving their valid address.

The Total disruption for 10 users was about 30 minutes.      

Lessons learned: Slow down with newer/unfamiliar software.

Reflection, Years Later

What stands out now isn’t the mistake. It’s the delay.

The dependency was there the entire time. The system just didn’t surface it. 

Deleting a network object also removed the DHCP scope, but the interface treated those as separate concerns. Nothing warned that a live service was tied to that entry.

The impact didn’t show up immediately because the system was still coasting on existing leases. Everything looked healthy until renewal time. Only then did the dependency make itself known.

That pattern has shown up repeatedly since then.

Modern platforms tend to collapse multiple services behind a single portal. 

The UI emphasizes structure and organization, but often hides runtime behavior. 

Actions that look administrative can have operational consequences, sometimes delayed, sometimes quiet.

What this incident reinforced is that dependency awareness is often retrospective. You learn what mattered after it stops working. The outage becomes the documentation the portal never provided.

Today, I approach unfamiliar systems assuming that dependencies exist even when they aren’t visible. If a tool makes destructive actions easy, I assume the blast radius is larger than advertised. And if nothing breaks right away, I don’t take that as proof that nothing was affected.

The outage was small.    The lesson wasn’t.

Some systems only explain themselves when they fail.

When Extra Hours Aren’t Extra Work

I had a job last week that should've been straightforward. The technical part? Easy. I've done it dozens of times.

The problem was I spent half the day waiting on someone else.

I was on site, ready to go. But every step forward required a remote tech who was juggling three other projects. I'd finish my piece, then wait. Get confirmation, move forward, then wait again. None of the delays were long enough to complain about individually, maybe 15 minutes here, 30 there, but they added up.

What should've taken four hours took seven.

When I submitted the invoice, the client pushed back on the hours. Fair enough, it looked like scope creep. But it wasn't. The work didn't expand. The plan just assumed the remote tech would be available when I needed them, and they weren't.

That's the part nobody wants to budget for: coordination time.

What I'm thinking about now:

Most job overruns aren't decided during execution, they're baked in during planning. Once you assume ideal availability and linear sequencing, you're already committed to the outcome. You just don't know it yet.

This keeps happening. POS rollouts, network projects, field service jobs, same pattern. We plan like resources will be there when we need them, then act surprised when reality doesn't cooperate.

I don't have it figured out, but I think it starts with planning for coordination, not just task time. If we're going to budget realistically, we need to account for the fact that most jobs depend on people who are doing three things at once.

DMVPN

DMVPN phases

Originally shared May 7, 2019. Updated for context and reflection.

Dynamic Multipoint VPN (DMVPN)  is a secure network that exchanges data between sites without requiring traffic to pass through an organization's headquarters' virtual private network (VPN) server or router.

Phase 1 — Hub & Spoke Only

In Phase 1, all spokes register with a central hub using NHRP. Traffic between sites must traverse the hub & spokes do not establish direct tunnels to each other. Routing is simple, but scalability and efficiency are limited for spoke-to-spoke traffic. TechTarget

Phase 2 — Dynamic Spoke-to-Spoke

Phase 2 introduces multipoint GRE at the spokes, allowing them to form direct tunnels with other spokes. Traffic no longer needs to transit the hub for every flow, improving efficiency. However, route information must be fully known at each spoke, and summarization isn’t practical in this case. networkjourney.com

Phase 3 — Scalable Shortcuts

Phase 3 keeps dynamic spoke-to-spoke tunnels and adds NHRP Redirect/Shortcut behavior. This lets spokes learn about each other’s next-hop addresses through the hub’s control plane, then establish direct tunnels more efficiently. Phase 3 also supports routing summarization, which helps reduce routing table size and improves scale. Cisco 

Looking Back

Back when I wrote this, DMVPN was simply the right solution for the problem in front of me. What became clear later is that DMVPN was also my introduction to the fundamentals behind modern SD-WAN, separation of control and data planes, dynamic path selection, and optimizing traffic without static complexity.

Understanding DMVPN made SD-WAN concepts feel familiar rather than abstract. While the tooling and terminology have changed, the underlying ideas haven’t. That’s been a recurring lesson throughout my career: technologies evolve, but fundamentals repeat.

Multicast Source Discovery Protocol (MSDP)

Originally written in May 2019 while studying and working through multicast design concepts. Revisited in 2025 to add perspective gained from real-world network design and operations.

During one of my Incredibly fun projects with BGP, I had to quickly and efficiently research, learn, and implement a new ( to me ) technology. While setting up my side of a BGP peer, I also had to set up a Multicast Source Discovery Protocol (MSDP) peer. Of course, the service provider was like Oh, and not forget the MSDP config. Because the service provider uses Juniper equipment, I started my search on the Juniper help pages.

The Multicast Source Discovery Protocol (MSDP) is used to connect multicast routing domains. It typically runs on the same router as the Protocol Independent Multicast (PIM) sparse-mode rendezvous point (RP). Each MSDP router establishes adjacencies with internal and external MSDP peers similar to the way BGP establishes peers. These peer routers inform each other about active sources within the domain. When they detect active sources, the routers can send PIM sparse-mode explicit join messages to the active source. 

The Juniper TechLibrary did a really great job of spelling it out for me. After that, the configuration was pretty smooth, except for a mistyped password. luckily, a quick look in the log showed an MSDP authentication error :)

blog.xuite.net/flytw1/

Looking Back

Looking back at MSDP, the problem it solved was a real headache at the time: sharing multicast source information across different domains without turning every hop into a mess. It was a very specific tool for a very specific problem, and it taught me two lessons that still apply today.

Design over features.
You didn’t use MSDP because it was “cool”; you used it because the architecture demanded it. It was never a silver bullet, just the right tool when the design called for it.

The “social” side of networking.
The hardest part of MSDP, and of modern SD-WAN and overlay designs, isn’t the configuration. It’s the coordination between teams and domains to make sure everyone shares the same understanding of how the network is supposed to behave.

MSDP isn’t the “hot new thing” anymore, but the core ideas behind it, source discovery and a clear separation between the control plane and the data plane, are the same ones showing up in modern fabrics and application-aware routing. Knowing why you’re using a protocol has always mattered more than memorizing the CLI.

What is a TACLANE

If you read my resume you will see that I work with “Tactical Local Area Network Encryption (TACLANE) devices”, or “Type 1 Encryptor “. TACLANES are High Assurance Internet Protocol Encryptor (HAIPE) Type 1 encryption devices that comply with the National Security Agency's (NSA) HAIPE IS (formerly the HAIPIS, the High Assurance Internet Protocol Interoperability Specification). I know that is a mouth-full of alphabet soup. What it means is these devices are typically used as secure gateways that allows two or more enclaves to exchange data over an untrusted or lower-classification network. The cryptography used is Suite A and Suite B, also specified by the NSA as part of the Cryptographic Modernization Program.

HAIPE IS is based on IPsec with additional restrictions and enhancements. One of these enhancements includes the ability to encrypt multicast data using a "preplaced key". This requires loading the same key on all HAIPE devices that will participate in the multicast session in advance of data transmission.

Unsolicited

Great and unsolicited

https://www.linkedin.com/feed/update/urn:li:activity:6504408961226199040

Subnetting

This post links to a subnetting walkthrough I built as a visual teaching aid. It reflects how I explained core IP concepts to others at the time.

Click this link ---------->      Fun With Sub-Netting

ADDing

Extra point to Melissa for finding the errors

Cisco Softphone register to a router in GNS3 running CME

I recently built a nice topology in GNS3. So I figured why not see if I could get one on the routers to run CME, and then get a Cisco softphone to register to the CME.
I connected the CME router to the laptops loopback and IPed the loopback and router interfaces. Then I installed the Cisco IP communicator (Soft IP Phone). During the softphone configurations, I put the router’s IP as the TFTP server IP.The 1st try was a No-Go ☹…. The CME I am running is very basic setup, with no phone loads.   I had to connect to a CUCM to get the Softphone to load. I re-configured the Softphone’s TFTP setting to point to the CUCM and it downloaded its needed files and register to the CUCM. I re-re-configured the TTFP setting pointing back to the router’s IP and …JOY 😊  

After the phone registered I had to add the mac-address to the ephone and configure a dn.
!
dn 111
!
ephone 3
mac-address 0026.b94f.adfb
button 1:1

Great CCNP LAB

This Lab was great fun to build. I started with the Multi-area OSPF then just keep on adding 

This project Started out with routers A0 and ABR-ASBR and A1 using OSPF.
Then adding router RIP and redistributing the rip routes at router ABR-ASBR.
Then adding router A10 with a Virtual-link.
Then adding routers R1-IPv6 and R2-IPv6 using OSPFv3.
Then Tunning IPv6 over IPv4 from A0 to R1-IPv6 across Tunnel 1 and using OSFPv3.
Then Tunning IPv4 over IPv6 from R2-IPv6t to R1-IPv6 across Tunnel 2 using RIP.

Tunnel 1 just uses local-link IPv6 addresses for the tunnel.
Tunnel 2 IPv4 addresses are not advertised.

Project title
Great CCNP-Lab
Author: KLiv                                          GNS3 file:  GNS3 2.1.6 Lab

CUCM SIP Trunk

Created a Test where one phone could call another on a different CUCM