Incident Report for Nov 5, 2008

2 views

Skip to first unread message

Marc Pope

unread,

Nov 7, 2008, 8:20:30 AM11/7/08

to falcon-...@googlegroups.com

November 7, 2008

Falcon Internet - Incident Report

Network Downtime - November 5, 2008

10:15AM EST: A major outage effected our network on Wednesday morning. We were notified by automatic page by our remote system monitor. After about 10 minutes of diagnosis, we determined that it was either our main switch or our main internet circuit. We dispatched to our data center while another issued a trouble ticket to twTelecom, our main internet provider.

10:30AM EST: A trouble ticket was entered at twTelecom for emergency network down and we waited for a technician to return our call. The person who takes the calls mentioned that the circuit was pending disconnection. She was mistaken as she was looking at our older circuit from Winston Salem which is being disconnected this week. (at the time, this was of great concern that they possibly disconnected the incorrect circuit!)

10:40AM EST: Our tech, Joel arrived at the data center. The link light to our main switch was out indicating no service from twTelecom.

10:42AM EST: A call was made to our account rep and to the plant manager for twTelecom but we were unable to reach them right away. We informed them of the possible mixup and demanded fast resolution.

10:45AM EST: twTelecom called and informed us the problem was on their side and they were working on tracking it down.

In the mean time, we were checking other systems and networks for related downtime in the area to see if it was isolated to us or a larger problem. We checked with three other companies that get their connectivity from twTelecom and they were not having any known issues. This included companies in the same building.

11:45AM EST: twTelecom tech called from St. Louis, MO and said the problem has been fixed. It was their fault. A fiber cross-connect was not seated properly and had come loose.

After the fact updates: I talked to my account rep and aparently the plant manager for twTelecom was pretty peeved that this happened and he was wanting some heads to roll. My account manager was somewhat confused that something in St. Louis caused my downtime, so we are trying to get to the bottom of that, we'll keep you updated!

At Falcon Internet, we know that network downtime can be critical to your business and we take it very seriously. While some things are our of or control, we take great steps to prevent this sort of thing from happening. We will continue to work toward minimizing future downtime with new and improved services as we go forward. We appreciate your business. If you have any concerns, please do not hesitate to contact us.