News Story

http://kdhnews.com/gallery/news/metroplex-wired/image_d1c90b94-0eb7-11e4-880b-001a4bcf6878.html?mode=jqm

Left a really Good Job

Changed jobs this week.

So technically I'm unemployed this weekend.

The old job was one of the most challenging and rewarding jobs I've ever had, and I've had a lot! 

I met and worked with some great networks and people.

This is what they said when I told them I was leaving.

This is the crew!

(They asked to join the witness protection program)

Cisco TAC to the Rescure

We Upgraded 12 Stacks of Cisco 3850 Switches today.

Which is no Big deal except...Cisco had to come and save the day. 

We copied over and installed the new 3.3.3 IOS which was supposed to Fix a bug in the 3.3.1 version we were running. All that was needed was a reboot, which I initiated a Change Control Request for, and which had been approved for 06:00 Monday, as it would require a small disruption. @06:00 Monday morning I was ready to reboot the 1st stack. The site staff had asked to reboot one stack first just to ensure the upgrade would be compatible and that the switches in the stack would come back up.  Always wanting to be the customer service driven professional that I am, I agreed. So after the 1st stack rebooted and came back up I checked the EtherChannel and the Logs, and once I notified the site staff, I rebooted the 11 other stacks of switches. All told at about 06:30 all 12 stacks were back up and by 06:45 I had logged into and checked all 12, with no issues noted. Or so I thought, because just before lunch we start getting email notification that some APs at this same site were disassociated from Controller and then rejoining just a minute later. After scouring the logs we noted that it was only a handful of 1200 series APs and that they were losing PoE and rebooted, we also notice an "Imax" error in the log for the interface just before the AP lost power and reboot. By lunchtime, I had not found anything pertinent on the web so I decided to call in the Big Guns, Cisco TAC. I used to think TAC was just for warranty, boy was I wrong! Cisco TAC will, of course, help you RMA a bad device, but I have used them more for figuring out how to configure a device or like this case figuring out a bug in the software. That is why Cisco calls it a SmartNet agreement, not just an extended warranty. After about an hour of getting the Cisco engineer up to speed, and a WebX, where he can see my screen and even drive if needed, he was able to pinpoint the bug AND get me a workaround. The issue was the older AP and the new IOS on the newer switches, and the workaround is to statically configure the MAX power for the PoE to 15.4

                                                                   Stack(config)#int g4/0/46

                                                                   Stack(config-if)#Des ***Old AP

                                                                   Stack(config-if)#power inline static max 15400

                                                                   Stack(config-if)#shut

                                                                   Stack(config-if)#no shut

These are older APs nearing EoL so it is not an Issue with our other site as we have replaced most of these APs.

THANK YOU Cisco!!

VSS

A Picture is worth a thousand words!

This is footage from a IDF Security Camera 

After we didn't plug the camera into the correct port. SiteOPs called me the next morning and asked if I knew these two yahoos.....

Greater Austin Area Telecommunications Network

Great Video about the GAATN in and around Austin

Troubleshooting 101

 Spent 2 ½ days trying to figure out why a customer's  VPN solution was not preforming the way the vendor had promised.  Never mind the fact that the networking team had no say in the installation or operation of the solution. Spent one whole day just tracking down the laundry list of issues the customer was having and any troubleshooting steps the vendor and customer had done. The second morning we traced the path checking routing and firewall ACLs. No issues, so I asked the customer if I could have a client to generate traffic on demand (and also check it’s setting). The customer was glad for any help and brought me a client. Checked firewall and IP setting, no issues. The third morning the vendor was a little upset that I was looking at the client, they were sure it was the network. After some back and forth, their tech had to go to lunch, and created an account on the server so I could do some tracert, ping, ect. 10 minutes late I had it figured out.

So, 1^st thing I did was a tracert to a connected client, and what do I see……………….

 I checked the server’s IP settings and the subnet mask was incorrect, as soon as it was corrected ALL the issues the customer had cleared up, even some they had but on the back burner.

The clients have a /21 bit mask, and the server was using a /24, the vendor’s tech swore up and down he had not changed it, and that it had been working up to that point. He even wrote an email saying “How strange it was that it stopped working now” I wrote back it was strange it had worked at all.

General troubleshooting steps

1.     Define the problem.

2.     Gather detailed information.

3.     Consider probable cause for the failure.

4.     Devise a plan to solve the problem.

5.     Implement the plan.

6.     Observe the results of the implementation.

7.     Repeat the process if the plan does not resolve the problem.

8.     Document the changes made to solve the problem.

MAC Spoofing to the rescue

Great Story from another one of my Students. 

In the CISCO class I am currently taking at CTC, with Mr. Cisco as my instructor, I have learned about the many protocols, applications and principles on how data travels over different networks. I have also learned the way people try to gain access to that data and the ways a network can be built to defend against those attacks. I never imagined that I would use the very thing that I have been trained to defend against for my own benefit.
This past weekend my son and I had it all planned to have some good ole father and son time by catching a movie, having dinner and then staying the night in a hotel playing Xbox until we pass out. Movie and dinner was great and it is now video game time. I hook up the Xbox to the TV in the room, turn it on and get ready for the fun to begin. The last step before the festivities kick off is to gain access to the internet. Houston we have a problem. Hotels, much like the local Mcdonalds and Starbucks, has a captive portal that requires you to agree to their terms and conditions and put in some basic information before I could access their internet connection. This requires a web browser that the Xbox does not have. I was stumped but the heartbroken look on my sons face did not allow me to give up. I referred to my training and understanding on how LAN's work and how data travels across the network. This is where MAC spoofing came to save the day.
I realized that I already set up my phone to connect to the hotels internet service and that if I could use that connection on my Xbox all will be good. I dug through the network settings on the Microsoft device and came across alternate MAC settings. Hallelujah. I took the MAC from my already accepted phone and applied it to the Xbox. This time when I tried to connect there were no issues and the online gaming session was able to commence. I was very proud of myself and of course I got my hero points from my son. It was very gratifying being able to use what I have learned for my professional career to solve a problem outside of work. MAC spoofing is the normally the enemy that I guard against, but that day it was my friend.

Excellent job David! 

dbl

Had to do some QoS today and ran across this dbl

Dynamic Buffer Limiting

Industry’s First Hardware and Flow-Based Congestion Avoidance at Wire Speed

A Cisco innovation, Dynamic Buffer Limiting (DBL) is the first flow-based congestion avoidance quality-of-service (QoS) technique suitable for high-speed hardware implementation. Operating on all ports in the Cisco Catalyst 4500 Series Switch, DBL effectively recognizes and limits numerous misbehaving traffic flows, particularly flows in a network that are unresponsive to explicit congestion feedback such as packet drops (typically UDP-based traffic flows). It is a multiprotocol technique that can examine a flow’s Layer 2/3/4 fields.
DBL provides on-demand Active Queue Management by tracking the queue length for each traffic flow in the switch. When the queue length of a specific flow exceeds its limit, DBL will drop packets or mark the Explicit Congestion Notification (ECN) field in the packet headers, so the flow can be handled appropriately by servers in the unlikely event of network congestion. Unchecked flows—also known as belligerent or non-adaptive flows—use excessive bandwidth, and their consumption of switch buffers results in poor application performance for end users. This misbehaving traffic can also negatively affect well-behaved flows and wreak havoc with QoS.

http://www.cisco.com/web/about/ac123/ac114/ac173/Q1-06/p_19.html

I know it sounds a little sale pitchy, but still worth looking at.

Shot myself in the foot today.

Actually I pulled the trigger yesterday, but didn't feel it till this morning.

So we have this "new" IP Address Management (IPAM) software (InfoBlox), which also does DHCP and DNS. Well yesterday, around 11:30am, I was in the IPAM section creating a new network.  I mistyped the Network address, and had to delete it out of IPAM. I must have highlighted the user’s network which checked its check box without realizing it.  Because this morning I received a bunch of calls that users at one site could not login this morning. I know DHCP was the issue; because the user’s IPs were 169.254.x.x/16.  I jumped on the switch and used the “sh ip dhcp snooping binding” to see if the any client had received addresses.

There were a few, but their lease times were old, we set our lease time to 1 day (86400 sec). This led me to check the DHCP server, where I did a search for the Network and found it missing! In this new software the IPAM and DHCP databases are connected, deleting the Network deletes the DHCP scope for that network. Of course the reason we didn’t get any call yesterday is because all the clients had already received the leases for the day and were go to good till this morning when they tried to renew their IP addresses. I rebuilt the Network and the DHCP scope, and the clients started receiving their valid address.

The Total disruption for 10 users was about 30 minutes.      

Lessons learned: Slow down with newer/unfamiliar software.

Cisco 4500X w/VSS

Well I followed the step laid out in the awesome article 

Virtual switching system (VSS) Config For Cisco 4500

Can't wait to put it into production.

A Mad Man's Playground 

             

They have an SD slot

And It Worked!!

THANKS FOR YOUR SERVICE!!!

Got this is the mail today, Kind of cool.

Got this is an email this week

Found this one in and old email 

Student's Cell Phone

Another one of My student's Cell Phone

One of my CCNA student's Windows Cell Phone with Putty on it, SSHing to his home Cisco  2960

Way to go CW! 

You can’t handle the truth!

You can’t handle the truth!

Son we live in a world that has networks. 
These networks have to be guarded by men with Certifications. Who’s going to do it, You, Your system administrator? 
You don't want the truth because deep down in places that you don’t talk about at parties; you want me on that Console, 
you need me on that Console!
I have a greater responsibility than you can possibly fathom. 
You weep for the users, and you curse the network. 
You have that luxury, you have the luxury of not knowing what I know, that Firewalls while tragic, probably saves data. And my existence while grotesque and incomprehensible to you, saves data.
 We use words like Subnet, ACL, OSPF, we use these words as the backbone of a life spent defending something, you use them as a punch line.
I have neither the time nor the inclination, to explain myself to a man, who rises and sleeps under the blanket of the very network that I provide, and then questions the manner, in which I provide it.

I'd rather you just say 'thank you' and go on your way. 

Otherwise I suggest you pick up a console cable, and stand a post. Either way, I don't give a damn, what you think you are entitled to!

(T Charping)

ASA Failover

Simple link upgrade from 1Gb to 10Gb,

Simple link upgrade from 1Gb to 10Gb,

SiteA Sw1 has a 1Gb link to Core Sw2. We want to upgrade it to a 10Gb link.

Easy enough right? Just copy the configs off ports G0/1 on the SiteA Sw1, and the Core Sw2

Then paste them on to ports TenG0/24 on each Sw.

Unplug the fiber at both Sws, and plug into the new ports.

(Now we did a lot of leg work before the upgrade.

We made sure the fiber was good and would support the TenGig

We installed the new optics)

So when we switched over the fiber and the ports lit up,

We just know we were good-to-go.

Buy when we tested the link by shutting down the port to Core Sw1.

…. Was all we got, no eigrp routes, but I could see Core Sw2 with Show CDP neighbors.

Of course this is not good, and I starting thinking what could be wrong.

A quick thinking colleague,  said “passive-interface” and sure enough because Site1 had

Access Layer ports so it was using default passive interface under the EIGRP config.

Core Sw2 was not because it did not have any Access Layer ports.

No default passive-interface t0/24 on SiteA Sw1 worked like a charm.

!!!! All day long.

Just goes to show you how not thinking about the big picture and a little command,

Can sneak up and bite you.

Quiet Weekend at home

Have a quiet Saturday at home this weekend. 

Swept some leaves off the front pouch, and had an emergency faucet repair. 

Then converted a physical computer to a virtual machine computer using VMWare’s P2V  standalone converter. Real easy to install and run on the physical machine, then just point it at the ESXi host, wait about 25 minutes and you have an exact duplicate of the physical computer in your virtual world.

Bandwidth Hog at a low bandwidth site with a little help from the "Bandwidth" Command

                   We have a remote site that has not yet been moved to our fiber transport ring, so it is on an aggregated (3) T1s link to the rest of our network. It is a small site with less than a handful of users, who only use the network to do their time cards, so 4.5Mb is fine for them normally, but the other day we receive notice, from our service provider, that the link has been saturated 24/7 since the beginning of the year. One thing about these aggregated WAN links is that the service provider handles the aggregation and passes you the combine link as an Ethernet link. 

      So what's the issue? 

How do I track down the big talker, if all it takes is 4.5 Mb to saturate the WAN link? 

          I ran a “sh int | i  /255” command to identify any ports that have high rxload or txload rates, BUT, because they are 10/100/1000 ports all their reading were 1/255. So I used the range command to set the bandwidth label on each interface, and  voilà, port 1/0/1 was receiving at 106/255 and port 1/0/48 (the up-link to the router) was transmitting at 06/255. I ‘m no rocket scientist, (but I did sleep at a Holiday Inn last night) but the traffic coming in on port 1/0/1 was leaving on port 1/0/48.

I found my big talker.

Nice and “simple” new IDF and equipment install?

Nice and “simple” new IDF and equipment install.  Featuring the Cisco 3850

It is great when we get a chance to install new equipment in a new IDF. No old equipment or old configurations to matchup or worry about.  Of course when the equipment is the new (new to me, that is) Cisco Catalyst 3850 Series Switches, with their Cisco StackWise-480 technology providing 480 Gbps of stack throughput, Packet Capture capability with the embedded WireShark, and  Stateful Switchover (SSO) resiliency, fun is on the horizon. The 3850 runs Cisco's IOS-XE Operating System (OS) which does looks and feels a lot like the old familiar IOS, but underneath the CLI, it is a whole different animal. So when you load a new OS here are some of the different commands:

·         Upgrading Cisco 3850 Stack IOS-XE :

A.      Copy the *.bin file to active member (you can copy it from a USB drive)

a.        You can use a USB drive, instead of TFTP if you what to, the USB port is on the front panel (at least on the 3850-48Ps)

B.      Use the "software install file flash:cat3k_caa-universalk9.SPA.03.03.xx.SE.150-x.xx.bin new" (make sure you get the "new" at the end of the line)

a.       **** You can load it from the USB drive if there is enough free space on the USB drive. “software install file usbflash0:cat3k_caa-universalk9.SPA.03.03.00.SE.150-1.EZ.bin new”***

C.      After the Switch copies the software, it isn't just one file anymore, it will ask for a re-boot

D.      ********13 minutes while the stack reloads

E.       Use the “software clean” command to clean un-user file in flash

Once the new OS is loaded and the switch is all good to go, make sure the switches are stacked if needed. I love the new stacking cables; they just seem to connect to the switch better the old type.

One of the changes with these new stacks is the way each stack will have an Active and Standby member to facilitate SSO resiliency which ensure the management plane is never unreachable. You can assign the Active and Standby roles to specific switches in your stack by setting the switch’s priority, the higher the better (max 15). We configure our uplinks in Ether-Channel groups with one port on the 1st switch in the slack and the other on the last switch in the stack, so we set the priories on these two switches in case we lose one the other will still have access to the management plane.

Speaking of Up-links, these new 3850s had uplink modules, right now there are three, a 4x1gb, a 4x1gb/2x10gb, and a 4x10bg. The 4x1gb/2x10gb module allows you to use all 4 port at 1gp, or 2 at 1gb and 2 at 10gp, and I have found (and Not in Cisco documentation) that when working with the 4x1gb/2x10gb module, it is best to disable the g1/1/3 and 4 interfaces, then enable the Te1/1/3 and 4 if you are using the 10Gb ports. Otherwise the TenGb port might err-disable and need to be re-enabled.  

The embedded WireShark is a very handy feature, But, and it is a big But, it is only available with the ipservices license, and that IS Not in Cisco Documentation yet either.  But if you are running the right Feature set, the syntax is different than what you may have used on the PIX and ASA’s.

1.       First configure an access-l to mark the interesting traffic

a.       access-list standard My-cap_acl

                                                               i.      It doesn’t have to be standard

b.      permit 198.214.208.24

c.       end

2.       monitor capture buffer My-cap_buff circular

a.      Creates a Buffer named My-cap_buff

3.       monitor capture buffer My-cap_buff filter access-list My-cap_acl

a.       associates the My-cap_buff to the My-cap_acl

4.       mon cap point ip cef My-cap_point g0/1/0.523 both

a.      Creates a capture point called My-cap_point

5.       mon cap point associate My-cap_point My-cap_buff

a.      Associates the My-cap_point to the My-cap_buff

6.       mon cap point start My-cap_point

a.      This starts the capture

7.       sh mon cap buffer My-cap_point p

a.      This show the capture parameters

8.       mon cap point stop My-cap_point

a.      This stop the capture

9.       And you can copy is off to a tftp server, or read it on the switch.

I am looking forward to use these switches and learning all their little tricks and nuances, and I just heard the 4500-X switches are here, And I might get to install them soon, check back for more fun.