When a storm threatens the availability of your operations, having an emergency response plan in place is critical. This preparedness was tested in October 2016 when Hurricane Matthew swept along Florida’s east coast. With torrential rain and winds forecasted at up to 110 mph, businesses were on high alert. Peak 10, a national IT infrastructure, connectivity and cloud solutions provider whose Fort Lauderdale and Jacksonville data centers were in the hurricane’s path, quickly put its Emergency Response Plan into effect to ensure the security, integrity and availability of its customers’ technology infrastructure no matter what Hurricane Matthew had in store.
Effectively Planning for Disaster 24/7/365 Environment
Storms the caliber of Hurricane Matthew, which began as a category 4 storm, can disrupt power, damage infrastructure and wreak havoc along their paths. To ensure availability in the face of disaster, Peak 10 puts considerable effort into advance planning. Using a thorough and systematic program that combines preventative maintenance, infrastructure monitoring, staff training and assessments, Peak 10 lays the groundwork to avoid or minimize the downtime and damage a crisis can cause.
“Upfront planning is the key to Peak 10’s service continuity plan,” explains Ken Swayze, compliance and security analyst at Peak 10. “We begin planning for emergency situations long before they are a reality.”
This preparedness begins with choosing its data center locations. Strategically located outside of flood zones, the geo-diversity of Peak 10 data centers allows customers to continue operations through other Peak 10 data centers if their primary data center becomes disabled. Tremendous care is also put into the structural capabilities of each data center. For enhanced storm protection, Jacksonville and Fort Lauderdale data centers boast hardened concrete construction and are designed to withstand flooding and hurricanes winds of up to 155 mph. Peak 10’s Fort Lauderdale data center also features Category 5 hurricane-resistant roofing, and Jacksonville is built to endure seismic activity.
To sustain an uninterrupted power supply, Peak 10 utilizes redundant UPS systems and generators – each with N+1 configurations. Multiple computer room air conditioning (CRAC) units with N+1 configurations maintain a sufficiently regulated IT environment if one cooling system goes down.
To further protect its critical systems and optimize performance and availability for its customers, Peak 10 employs a comprehensive preventative maintenance program. This regular maintenance is performed on key emergency systems including generators, UPSs, cooling systems, fire detection and suppression systems, and other critical systems to ensure they are ready when needed. Peak 10 also monitors its network and facility 24/7/365 to safeguard operations. This level of ongoing attention supports the efficient mitigation of issues.
Peak 10 also maintains relationships with outside vendors to guarantee backup plans are in place during a disaster – such as a hurricane. These partners include multiple Tier 1 internet providers to quickly transition between carriers without service interruption if one carrier goes down; multiple core network locations for further redundancy; and at least three fuel providers – two in-state and one out-ofstate – to service data centers during an extended outage. Partnerships are also in place with equipment vendors so failed units can be quickly replaced.
While Peak 10’s built-in redundancy, maintenance program and ongoing monitoring lay the foundation for continued operations during a disaster, having the appropriate plan in place is only half the battle.
“A plan doesn’t do any good if it doesn’t work,” notes Swayze. “It’s equally important to test that plan. This is a critical component of Peak 10’s service continuity planning.”
Equipped with highly tested storm and emergency preparedness plans and multifaceted recovery plans, Peak 10 is ready to minimize service disruptions and promote business continuity for its own operations. Its Emergency Response Plan outlines the policies and procedures needed to manage volatile situations. Peak 10 regularly tests its own readiness plan through live exercises conducted during normal operations.
These trials are handled by Peak 10’s “Go Team,” a geo-diverse group highly trained and experienced in network and data center operations and emergency response. This team participates in intense training to hone their preparedness for any situation. Cross-trained to perform multiple roles to ensure operational redundancy in an emergency, they can easily transition to any Peak 10 data center because of the uniformity of the Peak 10 network and infrastructure across data centers.
Peak 10 also ensures that the data centers are also stocked with food, water, bedding and other needed provisions to support the team and enable them to remain onsite as long as needed.
Ultimately, this level of preparedness positioned Peak 10 to successfully prepare for and weather Hurricane Matthew, while responding to and supporting its customer needs.
Efficient Processes Guide Preparations as the Storm Approaches
Peak 10 began monitoring Hurricane Matthew the week before it hit, and initiated its Emergency Response Plan. Using its Emergency Notification System, daily – and sometimes more frequently – communications were sent to customers to inform them that Peak 10 was prepared to address any storm-related issue. These communications also provided customers with updates on Hurricane Matthew, the specific preparations Peak 10 was taking and recommendations on how customers could protect their assets. To communicate with its internal team, Peak 10 also leveraged its Mobile Crisis Application to ensure its crisis management plan was readily accessible to all team members.
Armed with a preparedness checklist so no tasks were overlooked, Peak 10 began preparing its Fort Lauderdale and Jacksonville data centers for the storm by clearing the outside of the data centers of debris and securing doors and windows. Inside the data centers, UPSs and emergency standby generators were load tested; vendors were put on standby for emergency refueling; network and carrier connectivity redundancy were checked; contact information and communications plans were confirmed; and fuel tanks were topped off.
While Peak 10 adheres to a strict preventative maintenance cycle, unexpected issues can –and did – occur. Although a battery in one of the generators tested fine during its last maintenance cycle, a pre-storm review detected signs of failure, requiring the battery to be changed out.
Staffed with technical experts 24/7/365, Peak 10’s local Technical Assistance Centers (TAC) readied for the storm, and the Central Technical Assistance Center (CTAC), located in Charlotte, North Carolina, was put on alert to serve as the main coordination point for disaster assistance and control should the Fort Lauderdale or Jacksonville TACs become inoperable. To ensure the TACs were appropriately staffed to support operations, Phil Reissig, regional operations director for Peak 10, checked in with Fort Lauderdale and Jacksonville days before the storm to assess their staffing needs.
“I knew I only had a day or so to fly people in before the airports closed,” explains Reissig. “I called the data centers and asked if they needed any extra support. Jacksonville was all set, but Fort Lauderdale needed help.”
With almost 200 customers declaring disaster and checking into their disaster recovery (DR) suites, Fort Lauderdale was inundated with customers and needed additional staffing to support their needs. Reissig readied Peak 10’s Go Team to augment the Fort Lauderdale staff. Since all Peak 10 data centers are uniformly designed and utilize the same technology, the Go Team seamlessly transitioned into Fort Lauderdale operations without losing critical time to a learning curve. With additional employees on hand, Fort Lauderdale was equipped to support the influx of customers at the data center, assisting with everything from facility monitoring to troubleshooting to printing and faxing needs.
Outstanding Customer Service in the Face of Crisis
As preparations for Hurricane Matthew continued, Peak 10 was approached by the CTO of a financial solutions organization needing a DR suite and cabinet space immediately. The CTO visited the Fort Lauderdale facility at 10 a.m. the day before the hurricane. Peak 10 showed him an available DR suite and cabinet with in the data center, and within 30 minutes a contract was signed. Three hours later, the DR suite was cabled, the cabinet was fully powered and the new customer was fully provisioned to ride out the storm with Peak 10.
“Peak 10 was there when we needed them,” expresses the CEO of this financial solutions provider. “We needed a secure workspace to oversee our operations out of and Peak 10 was able to deliver that quickly under difficult circumstances.”
In fact, Hurricane Matthew served as a reality check for many businesses who did not have proven DR plans in place, driving an increase in inquiries after the storm passed.
Peace of Mind as the Storm Strikes
As Hurricane Matthew approached Fort Lauderdale Peak 10’s local staff made their way home to weather the storm with their families while the Go Team managed the data center.
A day after battering Fort Lauderdale with 87 mph winds, Hurricane Matthew swept through Jacksonville with winds up to 65 miles an hour, causing flooding along the coastline. However, due to its advanced preparations, the local Jacksonville TAC handled operations before and during the storm without any sustained damage to the data centers or storm-related downtime.
As the storm passed, Peak 10 personnel in both locations quickly returned to normal operations. An internal operational review was conducted by the Go Team to evaluate its processes and determine what could be improved upon or adjusted in the future. Go Team input and operations were documented and compiled in a report detailing the event and offering recommendations.
“Every situation is an experience to learn and build on what we already know,” summarizes Reissig. “Hurricane Matthew was an opportunity to put our existing emergency response plan to the test and it served us well. The bottom line is, Peak 10 is ready for whatever is thrown at us because we have prepared.”