In Case Of Emergency, Break Glass
Imagine that you’re the IT Manager for a mid-sized engineering company. You arrive at your place of work one morning to find emergency vehicles surrounding the smoldering ruins of what used to be your office. While you may be tempted to simply get back in your car and drive to the nearest pub where you can blissfully pretend it never happened, it would be far more productive to break out your disaster recovery plan and begin the process of rebuilding.
What is a Disaster Recovery Plan?
Put simply, a disaster recovery plan is a plan detailing how you’ll recover your business from disaster. Specifically, a DRP focuses on Information Technology, the computers that run your business. It doesn’t usually cover other aspects of your business, like supplier relationships, logistics, human resources, or non-IT facilities. A plan that encompasses all aspects of your business is referred to as a Business Continuity Plan. A DRP is just a sub-set of a BCP.
A DRP is not just a backup and restore procedure, although that’s a critical component. It also details what applications and data need to be recovered, the hardware required, contact information, scheduling, facilities, and more.
In recovering from a disaster, it’s necessary to set priorities. While eventually you’d want to get everything back up and running just as it was, not everything can or needs to be done at once. You define your priorities in terms of business functions. Let’s use our fictional engineering company as an example and think of the essential functions that must be resumed as quickly as possible.
Of course, you have your engineers who need to create and revise drawings. Accounting would certainly be considered an essential function as well. Without them, no one gets paid and nothing gets bought. But it may not be necessary to have the entire accounting department up and running immediately. Sales and Marketing? They can probably wait for a few days. The same may be true of Human Resources. At least some Purchasing may be required, but again, it may not be critical that the entire department is brought back online the first day.
Think of the hardware required for these functions. The engineers will need engineering workstations. Accounting and Purchasing will need basic desktop machines. They’ll all need printers and plotters, but perhaps one of each would suffice at this stage.
Now for your department, Information Technology. Some sort of infrastructure is required to support the other functions. That means servers. Do the engineers store their drawings on a file server? Do they have a database that keeps track of product data and revisions? Does Accounting have a financial database? Do they all have spreadsheets and other documents stored on a file server? These will all have to be restored.
E-mail has become a mission critical resource, so you’ll need that server. In order for the printers to work you’ll need a print server. Perhaps you also have an authentication server that controls access to the other devices.
To restore the data that resided on all of these systems, you’ll need your backup server and tape drives, as well as your most recent backup tapes.
Of course, you’ll need a local area network to connect all of these devices together. That implies network switches and cabling. E-mail will require some sort of Internet access. And don’t forget telephones and possibly a fax machine.
Not to be forgotten is a place to put all of this. Is part of the office building still serviceable? Could you temporarily relocate your computer room there? Or would it be better to rent a trailer that you can park on the site with network cables strung to the building?
Software and Applications
If your computer room was destroyed in the disaster, the new hardware you get to replace it will likely come blank, perhaps without even an operating system. It’s often possible to create bootable recovery discs, but those may only work correctly on identical hardware. Otherwise you’ll need copies of your OS discs so you can rebuild the servers. Once you’ve rebuilt the backup server, you may have to install the backup software before you can access your tapes.
If your server hardware is a close enough match, it may be possible to restore all required systems in a single step. If not, you’ll have to reinstall applications and then restore data.
Don’t forget the user’s software. Some companies back up servers, but not workstations. So you may need copies of the various workstation applications to restore those. How about your engineer’s CAD software and the accountant’s finance software?
People, the Key to the Plan
You know what needs to be done, and what you need to do it. Now, who’s going to do it? Do you have names and telephone numbers for everyone on your team? Remember, those are likely home and cell numbers; there may be no office phones at this stage. What about contact information for your suppliers? Who has the key to the safety deposit box where you’ve stored your off-site backup tapes? Who’s going to call the insurance company, the phone company, and your Internet service provider? Do you need a contact name at the local utility company to get power supplied to your interim computer room?
You may have the necessary phone numbers of people you need to contact, but can those people contact you? Make sure everyone involved – not just those on the core team – have the contact information they need.
Documentation, the DRP Itself
When disaster strikes, the initial reaction of most people is panic. Expect confusion. To counter confusion, you need a clearly defined and documented plan of action. This is the disaster recovery plan itself. Where is it? In the office that has just burned down? On one of the computers that has yet to be restored? Keep printed copies of the plan at secure off-site locations, and make sure several team members have access to them.
Besides what we’ve already discussed, here are some other components of a comprehensive plan:
- Copies of license agreements for purchased software in case you are later audited.
- Insurance policy numbers.
- Passwords and certificates for encrypted data.
- Software version numbers.
- Hardware serial numbers.
- Hardware descriptions and requirements.
- Recovery instructions for your backup system.
- A schedule detailing the functions that will be restored first, and those which come later.
There are resources available that can help you develop and execute a disaster recovery plan. IT consultants may work with you to define and document your requirements. Facilities exist that can provide co-location services for your entire infrastructure. They tend to be expensive to maintain, but they can have you back up and running at full capacity in a matter of hours.
Larger organizations with operations in geographically dispersed areas may find it effective to have redundant IT facilities. If one goes down for whatever reason, the other can take over at a moment’s notice.
A Never-Ending Process
Once you have your plan, make sure it works. If possible, test it at least once every two years. This may incur some cost, but it’s far preferable to finding out in the middle of an emergency that you’ve overlooked some key piece of information or forgotten a critical business function.
You also need to keep the plan up-to-date. People change positions, both within your organization and among your suppliers. Applications change. Hardware requirements increase.
All being well, you may never need your disaster recovery plan. But even if you don’t, having one may at least help you sleep better at night.