If your business isn’t completely dependent on technology, then you are in the minority these days. Given this dependence, protecting your business from an IT failure should be at the top of your priority list.
“Having been in the IT business now for 16 years, I’ve seen my fair share of close calls and, unfortunately, my fair share of outright disasters when it comes to IT,” says Zack Schuler, founder and CEO of Cal Net Technology Group. “There are three particular disasters that stick out in my mind. In each of these three cases, the companies were taking nightly backups of their data, and they thought this was enough.”
Smart Business spoke to Schuler about how businesses can avoid these kinds of mistakes.
What are some of the worst disasters you’ve encountered?
The first case was a company that had a sprinkler break right above its servers. They were taking a daily backup, however, they left the tapes on top of the servers, and the tapes were drenched and basically unusable after the downpour. The server hard drives were sent to data recovery, and after several days the company was up and running again. Had the tapes been taken off site, the downtime would have been significantly less, though they still would have had a day or two of being down, which in most cases is not acceptable.
The second case was a company whose building burned down. Their current tapes were stored on site; however, they had an older set that was taken off site. After a painful data reconstruction process, and several months later, they were able to get back on their feet.
The last case was a company whose Internet went down for a week, due to a major telephone company issue that had their T1 down. This was their only connection to the Internet, and their business was highly dependent on e-mail, so this outage had a significant impact on their business. They lost a percentage of their revenue as a result of the outage.
Needless to say, none of the above companies were prepared for the type of disaster that they suffered, yet all of them were backing up their data. I’ve seen too many companies think that a simple backup of their data, whether it be to tape, disk, or the cloud (a.k.a., the Internet), is sufficient.
How can businesses avoid costly downtime?
Here are three important questions that you can pose to whoever manages your IT, and some tips that will get you one step closer to being truly prepared in case of emergency.
1. What is your plan in case of a lengthy Internet failure? The smart thing to do is to make sure that you have multiple connections to the Internet, over different mediums. Having a connection via a T1 and a DSL line is not a smart move, as they both traverse over the strands of wire. An Internet connection through a telephone company and another through a cable provider is the way to go. This requires some complex routing and firewall work in order to maintain a seamless connection, but all of that will be well worth it when one of the two connections goes down.
2. What is your plan in case of a physical site failure, such as a fire, earthquake, etc.? Something as simple as a long-term power outage in your building can be a lot more common than one would think. On more than one occasion we’ve seen a building lose power for several days, and companies basically send their employees home. We had a client who was prepared in this scenario. They sent their employees to work from home, as they had a hot-site set up that employees were able to connect to from home and continue on with their work.
3. What is your plan in the event of a major hardware failure? Even if your equipment is under warranty, if a particular part fails on a server, and the vendor is out of stock on that part, you could see some downtime. I’ve seen this happen more than once. In this scenario, you should have a transition plan documented whereby you can easily move the data from this server’s backup over to another server, perhaps in a virtualized environment, to keep running.
What is the most common issue you’ve encountered with companies’ backup plans?
Perhaps the biggest overall error that I’ve seen companies make is that they don’t have any documented plan in place to recover from any of the above scenarios. Executives have been told that it’s all taken care of; their backups are off-site, and maybe they happen in real time up to a cloud provider utilizing state-of-the-art technology.
However, most companies simply don’t test their backups by going through a simulated failure. They assume that the backup is running, as they’ve been told. The No. 1 smartest action that you can take is to go through a simulated failure. Pretend that any of the above scenarios has happened, and try to recover from them. You’ll quickly realize that your IT department hasn’t thought everything through. Going through this exercise will be a great first step for them to figure out what they don’t know.
We assist IT departments with this type of work frequently, and we’ve never walked into a disaster recovery test whereby we didn’t make a tweak of some sort to make the plan better, thus more recoverable. Hiring an outside expert is certainly a great idea, but if you lack the resources to do so, another good move is to have one person in your IT department own the process of setting up the recovery plan, and then have another person in your IT department own the process of testing the recovery plan. If one person can test someone else’s work successfully, then you are well prepared.
Zack Schuler is the founder and CEO of Cal Net Technology Group. Reach him at [email protected].
