Confused as to where to start with regards to disaster recovery (DR) planning? We’ve recently published a DR planning checklist that you can populate with your company-specific data and be well on your way to protecting yourself in the event of a major issue. The document is great. However, when thinking about DR planning, I often think of this quote:
“In preparing for battle, I have always found that plans are useless, but planning is indispensable”
– Dwight D. Eisenhower
Over the years I have seen plenty of DR plans from companies. When reviewing them with the members of the teams that wrote them, we uncover a few commonalities that make the actual documents less than ideal. For one thing, people change — whether they change roles or leave the company entirely. Almost as soon as a document of this specificity is completed, it is out of date. Since DR is rarely a focal point for organizations on a daily or even annual basis, the likelihood of keeping this information up to date is not good. Even storing it in more collaborative places like a corporate Wiki or SharePoint site seems to do little to combat this eventuality.
Second, and far more common, is the dynamic nature of the technical environment itself. Dream organizations with little to no turnover can avoid problem one. But if they are successful and therefore in need of being brought back to life after a disaster, their technical environments are nonetheless ever changing. Whether entire systems are changed, server roles added or removed or rule sets modified, the end result is that what you document today won’t work tomorrow.
So should we give up? Of course not. In fact, let’s use another “Ike” quote to remove some of the doom and gloom and talk about how to proceed.
“Neither a wise man or a brave man lies down on the tracks of history to wait for the train of the future to run over him.”
- Dwight D. Eisenhower
We are all aware that stuff happens. History has shown us plenty of examples of this. My suggestion for someone just starting out with DR is to simply read the document and go through the questions with nothing more than mental answers. Visualize the who, what and where. Take several different disasters and think how you would react.
We’ve been conditioned to think of disasters as something either Mother Nature created or some catastrophic man-made situation that requires dozens (if not hundreds) of first responders to remediate. During my years in Florida, hurricanes were the topic du jour for the months May-September. Inevitably, most discussions started with “what if a hurricane is bearing down on us?”
It was a fun exercise to walk through, but I usually tried to steer the conversation to more likely scenarios such as hardware failures, data corruption or application issues. While not nearly as exciting, they are the issues that bring down companies more so than natural disasters. So read through the document twice. Once with each of the following scenarios in mind.
Scenario One – Purely IT-related Disaster
Remember the news recently where Starbucks had to shut down many of its stores due to an issue with their point-of-sale (POS) systems? For companies that do not have their customer base addicted to their product like moths to a flame, a situation like that could be the end. If your company is in a competitive marketplace where the options are aplenty and the ability to move from one provider to another is simple, an outage can drive people away in mass quantities.
Even after you bring your systems back online, the aftershocks will be felt on your bottom line for months to come. I doubt I am the only one who has switched cable providers because a software glitch that forced me to miss a key playoff game for the New York Rangers (causing them to subsequently lose the Stanley Cup, right?)
In thinking about this scenario, you don’t have to worry about things like: “Who goes where?” “Will I be able to call Joe or Susy?” Or “Will my alternate location be able to support my team?” Scenario one has your team at your disposal, most of your systems operational and no external factors to concern yourself with such as damage, safety and family. This is purely a technical problem requiring you to mobilize all of your technical resources as soon as possible.
Now your thoughts can stay around: “How do I communicate to affected customers?” “Do I have the capability to get all required resources together either virtually or physically to resolve the problem cohesively?” And “Are my escalation contacts with third-party software or hardware components current?”
Those will be the pieces of the document that will matter in this scenario. Small tip from years of scars: contact with your affected users is the single biggest area to have a handle on. A 30-minute outage with zero communication can be more damaging than a 1- hour outage with constant communication.
Scenario Two – Natural Disaster
Now we are reading through the template document again, but instead of ignoring all the larger logistical questions, you need to think about seemingly small things that pop up and cause big issues. What happens if there is no cell service? How is everyone going to get connected? If you need to leave the area (and assuming your equipment is in the same geographic area as you are) do you need to transport physical gear?
What about people and their more important concerns of family and property damage? Can you really count on your employees to leave the area and help setup shop many miles away? As much employee loyalty as you think you might have, it all goes out the window if someone’s family is in need.
Those factors, along with the de facto myriad of technical questions around data and connectivity, all need to be thought of properly. The best DR plans for surviving a natural disaster involve not having worry about who goes where, but rather whether the secondary site is online or not. If your DR plan can be as simple as making the declaration to swing traffic, you are (as Mr. Sheen would say) “winning”.
Not to gloss over the complexity that is required for such a setup, but the more you don’t care about the questions mentioned above, the better off you are. Your plan shouldn’t be figuring out how to solve all those questions, but instead how you can make them almost irrelevant.
So plan in your head repeatedly. If you can get all the information down on paper, great. But the mental planning is (as Ike might have said) the more important exercise of the two.