Greater Austin Area Telecommunications Network

Great Video about the GAATN in and around Austin

Troubleshooting 101

 Spent 2 ½ days trying to figure out why a customer's  VPN solution was not preforming the way the vendor had promised.  Never mind the fact that the networking team had no say in the installation or operation of the solution. Spent one whole day just tracking down the laundry list of issues the customer was having and any troubleshooting steps the vendor and customer had done. The second morning we traced the path checking routing and firewall ACLs. No issues, so I asked the customer if I could have a client to generate traffic on demand (and also check it’s setting). The customer was glad for any help and brought me a client. Checked firewall and IP setting, no issues. The third morning the vendor was a little upset that I was looking at the client, they were sure it was the network. After some back and forth, their tech had to go to lunch, and created an account on the server so I could do some tracert, ping, ect. 10 minutes late I had it figured out.

So, 1^st thing I did was a tracert to a connected client, and what do I see……………….

 I checked the server’s IP settings and the subnet mask was incorrect, as soon as it was corrected ALL the issues the customer had cleared up, even some they had but on the back burner.

The clients have a /21 bit mask, and the server was using a /24, the vendor’s tech swore up and down he had not changed it, and that it had been working up to that point. He even wrote an email saying “How strange it was that it stopped working now” I wrote back it was strange it had worked at all.

General troubleshooting steps

1.     Define the problem.

2.     Gather detailed information.

3.     Consider probable cause for the failure.

4.     Devise a plan to solve the problem.

5.     Implement the plan.

6.     Observe the results of the implementation.

7.     Repeat the process if the plan does not resolve the problem.

8.     Document the changes made to solve the problem.

MAC Spoofing to the rescue

Great Story from another one of my Students. 

In the CISCO class I am currently taking at CTC, with Mr. Cisco as my instructor, I have learned about the many protocols, applications and principles on how data travels over different networks. I have also learned the way people try to gain access to that data and the ways a network can be built to defend against those attacks. I never imagined that I would use the very thing that I have been trained to defend against for my own benefit.
This past weekend my son and I had it all planned to have some good ole father and son time by catching a movie, having dinner and then staying the night in a hotel playing Xbox until we pass out. Movie and dinner was great and it is now video game time. I hook up the Xbox to the TV in the room, turn it on and get ready for the fun to begin. The last step before the festivities kick off is to gain access to the internet. Houston we have a problem. Hotels, much like the local Mcdonalds and Starbucks, has a captive portal that requires you to agree to their terms and conditions and put in some basic information before I could access their internet connection. This requires a web browser that the Xbox does not have. I was stumped but the heartbroken look on my sons face did not allow me to give up. I referred to my training and understanding on how LAN's work and how data travels across the network. This is where MAC spoofing came to save the day.
I realized that I already set up my phone to connect to the hotels internet service and that if I could use that connection on my Xbox all will be good. I dug through the network settings on the Microsoft device and came across alternate MAC settings. Hallelujah. I took the MAC from my already accepted phone and applied it to the Xbox. This time when I tried to connect there were no issues and the online gaming session was able to commence. I was very proud of myself and of course I got my hero points from my son. It was very gratifying being able to use what I have learned for my professional career to solve a problem outside of work. MAC spoofing is the normally the enemy that I guard against, but that day it was my friend.

Excellent job David! 

dbl

Had to do some QoS today and ran across this dbl

Dynamic Buffer Limiting

Industry’s First Hardware and Flow-Based Congestion Avoidance at Wire Speed

A Cisco innovation, Dynamic Buffer Limiting (DBL) is the first flow-based congestion avoidance quality-of-service (QoS) technique suitable for high-speed hardware implementation. Operating on all ports in the Cisco Catalyst 4500 Series Switch, DBL effectively recognizes and limits numerous misbehaving traffic flows, particularly flows in a network that are unresponsive to explicit congestion feedback such as packet drops (typically UDP-based traffic flows). It is a multiprotocol technique that can examine a flow’s Layer 2/3/4 fields.
DBL provides on-demand Active Queue Management by tracking the queue length for each traffic flow in the switch. When the queue length of a specific flow exceeds its limit, DBL will drop packets or mark the Explicit Congestion Notification (ECN) field in the packet headers, so the flow can be handled appropriately by servers in the unlikely event of network congestion. Unchecked flows—also known as belligerent or non-adaptive flows—use excessive bandwidth, and their consumption of switch buffers results in poor application performance for end users. This misbehaving traffic can also negatively affect well-behaved flows and wreak havoc with QoS.

http://www.cisco.com/web/about/ac123/ac114/ac173/Q1-06/p_19.html

I know it sounds a little sale pitchy, but still worth looking at.

Shot myself in the foot today.

Actually I pulled the trigger yesterday, but didn't feel it till this morning.

So we have this "new" IP Address Management (IPAM) software (InfoBlox), which also does DHCP and DNS. Well yesterday, around 11:30am, I was in the IPAM section creating a new network.  I mistyped the Network address, and had to delete it out of IPAM. I must have highlighted the user’s network which checked its check box without realizing it.  Because this morning I received a bunch of calls that users at one site could not login this morning. I know DHCP was the issue; because the user’s IPs were 169.254.x.x/16.  I jumped on the switch and used the “sh ip dhcp snooping binding” to see if the any client had received addresses.

There were a few, but their lease times were old, we set our lease time to 1 day (86400 sec). This led me to check the DHCP server, where I did a search for the Network and found it missing! In this new software the IPAM and DHCP databases are connected, deleting the Network deletes the DHCP scope for that network. Of course the reason we didn’t get any call yesterday is because all the clients had already received the leases for the day and were go to good till this morning when they tried to renew their IP addresses. I rebuilt the Network and the DHCP scope, and the clients started receiving their valid address.

The Total disruption for 10 users was about 30 minutes.      

Lessons learned: Slow down with newer/unfamiliar software.

Cisco 4500X w/VSS

Well I followed the step laid out in the awesome article 

Virtual switching system (VSS) Config For Cisco 4500

Can't wait to put it into production.

A Mad Man's Playground 

             

They have an SD slot

And It Worked!!