They’re erecting buildings upside down in Boston.
In anticipation of sea levels rising at least two feet in the next 35 years, the first floor of the new Spaulding Rehabilitation Hospital at the Charlestown Navy Yard is three-and-a-half feet above the current 100-year flood level. There are no critical patient services on that floor. Exterior vents are placed high on the building’s walls. All the mechanical and electrical systems are on the roof. The hospital is designed to remain operational during and after a serious coastal storm like Hurricane Sandy.
This is a model building in the context of business continuity management (BCM) planning; that is, ensuring business resilience before, during and after an operational disruption. BCM includes IT disaster recovery management as well as so much more: supplier management, crisis management, emergency management, business recovery, contingency planning and preparedness.
Many tend to lump together business continuity (BC) and disaster recovery (DR). You can have a DR plan without BCM, but BCM without DR would be, by definition, not BCM. By extension, a DR plan should focus only on the recovery of IT services and none of the other elements of BCM, which many companies do tend to include. The narrower the scope and more detailed the instructions, the better the chances of a successful DR implementation. Ideally, the DR plan is one of a set of standalone BCM documents, each mutually supportive, but with clearly defined objectives and specific actions targeted at a specific audience or individual role (not person).
The Case for Everyday DR Planning
Given the widespread devastation that accompanies major natural disasters, these events are often top of mind when the need for DR planning comes up. Statistically, however, they account for only about five percent of downtime incidents. Hardware failures alone comprise more than one-half of disasters for small- to mid-sized businesses (SMBs). Next in line is human error at 20-plus percent. Software malfunction, much of it resulting from inadequate patch management and testing, rounds out the threat scenarios that can take your business down. Those ill equipped could expect up to 40 hours of downtime and recovery time.
Most natural disasters are not national news. Last February, accumulating snow collapsed a roof in Jersey City, NJ, damaging a Verizon building and causing a large section of cinderblock wall to give way. Fortunately, most workers were out on jobs when the collapse occurred. About a dozen trucks inside the building were damaged beyond repair, and some of the employees’ personal cars were crushed. Utility crews responded to the scene to repair a water main leak caused by the collapse. Is this a case of human error (not clearing the roof) or natural disaster?
An increasingly important reason for DR planning is regulatory compliance. In the financial and healthcare industries, in particular, DR planning and implementation are required and subject to audit. We know of one bank that failed its audit because its DR site was too close to the production site … 20 miles away. The Gramm Leach Bailey Act (GLBA) requires companies in the financial services industry to protect client data. Protecting private personal data includes the need to be able to access and provide that data essentially on demand, as well as knowing precisely where it is stored.
Introduce DR-as-a-Service into BCM
Disaster recovery as a service (DRaaS) is a leading growth segment in the cloud industry, for good reason. It is especially useful for SMBs that lack the necessary expertise to provision, configure and test an effective DR plan. It’s their own backup DR site but without the expense of their own facility, hardware and storage, security systems, physical infrastructure, or staffing. It is monitored 7/24/365 by trained technical staff. DRaaS contracts can be flexible as the business’ needs change. With the right provider, it can be a secure and fully compliant data center operation.
“A customer recently put its entire IT production environment in the Peak 10 cloud,” said Steve Harris, senior vice president. “They added our Recovery Cloud for DR. Typically, a DR site is a mirror or a very robust subset of the production site. In this case, the Recovery Cloud was about one-third the cost of the production environment. That cannot be realistically replicated outside the cloud.”
In the event of an actual disaster, an offsite vendor will be less likely than the enterprise itself to suffer the direct and immediate effects, allowing the provider to implement the disaster recovery plan even in the event of a total or near-total shutdown of the affected enterprise. Even if the vendor site is affected, choosing a provider with multiple geographically dispersed data centers dramatically lowers risk.
The increase in hybrid cloud adoption by SMBs is also a factor in the growth of DRaaS. One of the major drivers is the need for flexibility among user organizations. Cloud-based services are more flexible than traditional DR services; this enables end users to concentrate more on organizational core activities than on managing infrastructure and applications. Further, enterprises are able to establish a rapid deployment model that enables applications to be scaled quickly to match increased usage requirements.
Things to Watch Out For
Whether traditional or cloud-based DR, there are a few things that must not be overlooked.
Foster a culture that includes disaster recovery awareness and preparedness. Train new employees on recovery procedures.
Minor discrepancies, omissions and oversights in an organization’s DR plan can have a major impact on the time required to recover from a disaster and the associated business impact. Further, maintaining consistency between production and recovery environments remains one of the biggest DR testing and exercising challenges.
Test your plan frequently. A periodic walk-through of the procedure with the recovery team will assure that everyone knows their roles. Eventually, a component-level restoration of your largest databases will be needed to get a realistic assessment of your recovery procedure. Regularly test the systems to be used in recovery to validate that all the pieces work. Always record test results and update the disaster recovery plan to address any shortcomings.
As your business environment changes, so should your DR plan. Reexamine the plan every year on a high level. As applications, hardware, and software are added to your network, they must be evaluated for their potential impact on operations in the event of disaster and incorporated into the plan. DR should never be an after-thought, but a mandatory component of planning.
Out of sight, out of mind. Using DRaaS has many benefits, but don’t assume that once it’s in place it is no longer in need of regular review. Business dynamics must be reflected in the DR plan, as must the constant barrage of new business threats. Be sure the supplier maintains a robust, secure and audited infrastructure.