We Upgraded 12 Stacks of Cisco 3850 Switches today.
Which is no Big deal except...Cisco had to come and save the day.
We copied over and installed the new 3.3.3 IOS which was supposed to Fix a bug in the 3.3.1 version we were running. All that was needed was a reboot, which I initiated a Change Control Request for, and which had been approved for 06:00 Monday, as it would require a small disruption. @06:00 Monday morning I was ready to reboot the 1st stack. The site staff had asked to reboot one stack first just to ensure the upgrade would be compatible and that the switches in the stack would come back up. Always wanting to be the customer service driven professional that I am, I agreed. So after the 1st stack rebooted and came back up I checked the EtherChannel and the Logs, and once I notified the site staff, I rebooted the 11 other stacks of switches. All told at about 06:30 all 12 stacks were back up and by 06:45 I had logged into and checked all 12, with no issues noted. Or so I thought, because just before lunch we start getting email notification that some APs at this same site were disassociated from Controller and then rejoining just a minute later. After scouring the logs we noted that it was only a handful of 1200 series APs and that they were losing PoE and rebooted, we also notice an "Imax" error in the log for the interface just before the AP lost power and reboot. By lunchtime, I had not found anything pertinent on the web so I decided to call in the Big Guns, Cisco TAC. I used to think TAC was just for warranty, boy was I wrong! Cisco TAC will, of course, help you RMA a bad device, but I have used them more for figuring out how to configure a device or like this case figuring out a bug in the software. That is why Cisco calls it a SmartNet agreement, not just an extended warranty. After about an hour of getting the Cisco engineer up to speed, and a WebX, where he can see my screen and even drive if needed, he was able to pinpoint the bug AND get me a workaround. The issue was the older AP and the new IOS on the newer switches, and the workaround is to statically configure the MAX power for the PoE to 15.4
Stack(config)#int g4/0/46
Stack(config-if)#Des ***Old AP
Stack(config-if)#power inline static max 15400
Stack(config-if)#shut
Stack(config-if)#no shut
These are older APs nearing EoL so it is not an Issue with our other site as we have replaced most of these APs.
THANK YOU Cisco!!
