Based on a True Story: No Fairy Tales in this IT Department
The real world can be a harsh and unforgiving place, even for seasoned experts. It’s full of uncertainty, variables and legacy environments that will not be denied their foibles and oddities. When things do not go as expected (or promised), the resulting fire drills sap valuable resources and potentially disrupt “normal” operations.
Even Peak 10 – with infrastructure and process documentation managed by a full spectrum of technology experts using best practices – hits a pothole now and then.
One such instance occurred when we were working to configure critical new resources to play nice with other parts of our infrastructure. The vendor’s documentation methodically laid out the steps to be followed to make it all work. The results, however, were random unexplained outages on some compute nodes and resource contention. Only four customers, including Peak 10, had ever experienced this before. Because Peak 10 was one of the vendor’s bigger and more successful customers, it was there to work through the issues with us.
Know your clouds. Download the eGuide: Buyers’ Guide to the Cloud
This took every resource at our disposal and approximately three months to resolve. Peak 10 staff took turns working around the clock to devise fixes, install, run, uninstall and try again. The vendor made its support channels available continuously for an entire week. We redesigned the recommended configuration on the fly, akin to changing out a transmission while the car is running, as one Peak 10 engineer explained it.
At the end of it all, our supplier’s stated maximums for that particular implementation were far too aggressive. Instead of taking one big bite, the optimum solution was to create several smaller distributed domains to achieve the results we wanted and the services our customers needed. To accomplish it all without any of our customers being impacted during the process was no small feat.
The post mortem on this experience concluded with these observations. Just because something is on paper doesn’t make it factual. Just because you can do something doesn’t mean you necessarily should. And, importantly, rarely does one size fit all.
Security matters. Download Information Security – Best Practice: Invest in People
There is something else… and it is not always appreciated until a crisis occurs —the value of experience. Peak 10 had an EXPERIENCED, stellar IT staff working around the clock to solve a problem that had nothing to do with our company as the service provider. Instead, it was all about knowing what to do because we had firsthand experience working regularly with a full stable of clients with giant workloads.
When you are a business that is focused on doing its own business best, whether it is banking or gaming, your organization’s IT department is only used to dealing with its own in-house issues. Your IT department may hire a staff member or two with experience from another company’s in-house IT department, but it is rare to get the kind of experience that comes from working with a wide variety of organizations. Imagine having an IT staff that works with thousands of different types of companies and workloads and the kind of experience that welds. The best part? You don’t have to imagine it. It’s real —and it is at Peak 10.
Watching our great staff work together in a crisis makes me proud to be sure, but it also made me feel honored. How often do you get that feeling with your IT department?