You have to admire the tenacity and human spirit of the Americans when coping with events like Superstorm Sandy.
As lower Manhattan was flooded and the local power station was knocked out many of the datacentres in the region were put out.
InterNAP's data centre was evacuated and a customer note sent by Matt Price, the company's director of hosting operations and support, said the site's diesel pumps were out of action.
With no fuel available to the generators on higher levels they decided to call it a day.
“Life safety is our number one priority,” Price said. “We are making plans to completely exit the facility.”
Customers were warned to shut down servers immediately. Which is fair enough. Who can blame them? But another service provider, Peer 1, seemed to be made of sterner stuff. Its staff volunteered to keep the system going at any cost and physically carried buckets up diesel up 17 floors to the feed the generators that were to keep Peer 1's Manhattan datacenter in action. Seventeen floors! That's dedication.
The staff in some UK datacentres that I could mention can't even be raised to put labels on all the cables running onto the servers! So while some service providers can 'work on through the storm', there are others that won't survive an attack by a confused cabling installer.
Under circumstances like these the SLAs of the hosting providers come in for some scrutiny, and already there are signs of dissent from end-users. Clients of Internap, such as NuclearFallout Enterprises and chat service Zopim, posted updates on their service outages on Facebook and Twitter.
Others took to forums to complain. “It's rather sad that datacentres are promising fail-safes that don't actually work,” complained one, on Webhostingtalk.com.
Another poster on the same forum said: “Don't bother mentioning SLAs. Those are the hosting world equivalent to used toilet paper. Totally useless.”
“What I find frustrating is all the people who want to defend shoddy planning,” they said. “It's not like they didn't have a week to prepare.”
The problem is, you can never really anticipate what is going to happen until it happens. Then you need the right people with the right attitude to adapt. It's only by getting the right staff in place that service providers can guarantee they will keep going at all costs.
This is not something that is apparent when a service contract is out to tender, says Peer 1's UK MD Dominic Monkhouse. “Tenders are all about who can do the best speeds and feeds at the right price,” he sayd, “but they don't cover what you need to know.”
What you need to know is the attitude of the staff; happy staff will be motivated to keep going, but companies that hire and fire at will are unlikely to work.
So maybe, next time you are involved in the tender process for a service contract, it's time to pay attention to staffing details. Is the company in the Top 100 Best Places to Work? How fit are the employees? Maybe you should make them carry a bucket of diesel up 17 flights.
Image credit: Creatas
This was first published in November 2012