The Ready Big Project.
This Project had our team upgrading the networking equipment of one of our main sites, in preparation of our IP Telephony roll out. This site houses one of our two UC clusters, and facilities three of our largest and most visible departments. The project included replacing the Cisco 3560E Layer 3 switch with a Cisco 6509, replacing a Cisco 3550 switch with a Cisco 3850, and adding a Cisco 3750X to an existing 3750X stack.
Due to the 24/7 nature of the site the work had to be scheduled after hours and any loss of connectivity had to be kept to a minimum and a total maintenance window of three hours.
A lot of planning went into ensuring the ports from the 3560E to the IDFs were mapped correctly on the 6509. Most of the optical transceivers were reused, but all of the fiber jumpers had to be replaced with longer lengths. Furthermore the new Ten Gigabit optics in the new 3850 meant an optic in the 6509 had to be changed and of course the jumper connecter type had to be changed as well. Additionally we had to add an optic and jumper for the additional 3750x.
Once the 6509 was powered up and the configuration checked we broke the Ten Gigabit transport ring and connected the 6509’s 1st Supervisor’s Ten Gigabit port to the uplink while leaving the ring’s downlink in the 3560E so as we moved the IFD’s links the down time was minimized. To minimize any down time due to the boot-up of the 3850 one of the power cords was ran thought the rack to supply power to the switch so it would be booting while it was being mounted. After we mounted, cabled the stack cables, and powered up the new 3750x to the existing stack, we had to configure the “new” stack with an EtherChannel to the 6509. Once the EtherChannel came up we notice stack port 1 flapping, we reseated the stack cable on port 1, which fixed the issue. As we moved the IDFs off the 3560E and into the 6509, we tested connectivity to the IDFs with the Show CDP and Show Interface Trunk CLI commands.
After all the IDFs were patched to the 6509, all IDFs were tested to ensure clients were receiving DHCP addresses as well as logging into the equipment in the IDFs remotely. Once the 3570x stack came up and the stack port issue was addressed, numerous extensions were called and they were checked in CUCM to ensure they had registered with the correct UC cluster.
We did run into a few issues with this project, mostly with the count and types of fiber jumpers. As all the jumpers needed to be replace and some were Single Mode and others Multi Mode and with the addition of the new optics and jumpers, I lost count of the jumpers by type and did not bring enough of the correct type. This issue was fixed with a quick trip to the warehouse to pick up additional jumpers. Another issue appeared the next day as users started logging on. Users reported a scarcely use application was not working. The app used multicast and we realized when the config had been copied to the 6509 the IP multicast command syntax was different on the 6509 and did not take, and was not caught. The correct command was issued on the 6509 and because the IP PIM commands were on the interfaces the app start working.
Lessons learned on this project are, always have another set of eyes double check the inventory list to ensure counts are correct. Also check command syntax when coping configs between different platforms.
This was a really big project with a lot of moving parts. The entire project went well with very little disruption in service, and all work was completed in the maintenance window. Even with the issues, which were resolved quickly, we laid the foundation for the rest of the equipment upgrades and roll out. The team worked well together, and we learned about each other’s strengths.
