Sandy Washed Away Contingency Plans Across New York Region
Internet sites were kept online only because an impromptu bucket brigade ferried 5-gallon jugs of diesel fuel up 17 stories to a rooftop generator. Billions of dollars of stock trades were frozen and sidelined. Even the United Nations closed its complex when flood waters knocked out power.
Some of the world’s largest companies had contingency plans but in many cases they could not withstand the fury of the raging waters that poured into parts of the five boroughs that make up New York City.
“Most companies have a disaster recovery plan or business continuity plan but never test them or practice them,” said Vincent Renaud, vice president at the Uptime Institute, a New York-based research group focused on digital infrastructure.
“Mission critical infrastructure equipment – engine-generators, fuel storage, pumps, UPS, cooling systems – need to be placed out of harm’s way,” he said. “No one ever expected to have such a disaster on their hands like Sandy in New York City. Consequently, no planning for this level of catastrophe was done.”
Media mogul Barry Diller’s IAC/InterActiveCorp, which owns Newsweek Daily Beast, Ask.com, Urban Spoon and other popular websites, had to close its headquarters when the flood waters from the Hudson River entered the building, which lost power during the storm.
“We are working to restore all building operations,” an IAC spokeswoman said on Friday. “Most employees have been working remotely.”
While IAC’s properties remained online, among the more noticeable website crashes was the Huffington Post, whose parent AOL Inc relied on a back-up center in Newark, New Jersey, about 11 miles (18 km) from its main office in Manhattan. All three of the telecommunications providers serving the Newark location had outages the night Sandy hit.
“If I were Arianna Huffington, I would be as diverse geographically as possible,” said Gartner analyst Akshay Sharma, adding that Chicago would be a safer back-up location.
At Verizon Communications Inc, one of the top U.S. telecommunications companies, a giant switching station at 140 West Street in the financial district was knocked out by Sandy’s floodwaters. The company had moved its backup generators out of the basement after 9/11, but left under street level the tanks of fuel and fuel pumps. So when they were soaked, downtown New York communications were ravaged.
David Doddridge, who runs a construction and building code consulting firm in New York City, said fuel for rooftop generators is usually stored indoors and below street level for both practical and safety reasons. Fuel delivery trucks rely on gravity to load tanks, and roof-top fuel storage would raise concerns about lightning strikes and weather-related corrosion.
“The basement is really the most practical spot,” Doddridge said. “Not every scenario can fit every situation. From a logistics standpoint, fuel storage is typically in the basement. And it’s much safer to keep these inside.”
While Sandy was dubbed the frankenstorm because it was an unusual combination of a tropical disturbance and a colder system moving down from the north, scientists had been warning for years that a major hurricane could sweep up the Eastern Seaboard and deliver an overwhelming flood surge to New York.
Malcolm Bowman, a professor of oceanography at the Marine Sciences Research Center at the State University of New York at Stony Brook, wrote in a 2009 paper that a so-called 100-year flood could overrun parts of lower Manhattan, Hoboken in New Jersey, Staten Island and New York’s La Guardia Airport.
“It was just a matter of time,” Bowman said after Sandy hit. “It was all almost prophetic.”
He argued that the region’s piecemeal approach to address flooding needs to be replaced by a broad system of surge barriers or levees similar to those that defend low-lying European cities. Protecting individual facilities is “sensible and needs to be done,” he said. “But at the end of the day, it doesn’t protect you against a mega storm’s knockout punch.”
Experts said companies have to weigh preparing for storm risks against other disaster scenarios, such as a terrorist attack or an uncontrolled influenza epidemic.
“Scary predictions appear all the time – sorting them through beforehand is tough,” said Ernest Sternberg, chairman of the department of urban and regional planning at the University of Buffalo’s school of architecture, who has studied New York’s disaster preparedness.
“One must watch out for and prevent ‘hindsight bias’ in which we rashly judge people acting under great uncertainty when they are making decisions about future disaster possibilities.”
Some contingency plans never got a chance. On Wall Street, the New York Stock Exchange initially planned to switch trading to an all-electronic platform in case of a weather emergency. But NYSE Euronext reversed that decision after some traders said they were not prepared for the plan.
Many major trading firms, including Goldman Sachs Group Inc , CME Group Inc’s NYMEX, and Citigroup Inc, were in the flood-prone evacuation zone, which did indeed flood and as of Thursday still had no power.
After the NYSE resumed trading on Wednesday, some firms still had trouble. Knight Capital Group in Jersey City, one of the biggest brokers for retail stock trading and exchange-traded funds, shut down its platform for half a day when fuel supplies for its back-up generator ran low.
Knight has a backup facility in Purchase, New York, about 30 miles (48 km) north of New York City and out of harm’s way, but did not feel confident about switching over its service in the middle of the day. The firm asked clients to trade elsewhere “out of an abundance of caution.”
Many companies have generators, fuel or pumps in the basement or at ground level in the flood zone. Others had only enough fuel to last a couple of days, having failed to anticipate a power outage lasting a week or more.
“It’s liable to be up and down for the next week,” said James Cowie, co-founder of Renesys, which measures Internet performance. “There are people now revisiting their best practices, a lot of people who did not do adequate tests.”
Web-hosting service Peer1’s emergency generators were on the roof, but its basement fuel tanks were flooded. The Vancouver-based company scrambled and came up with a creative solution: It found new fuel in 50-gallon drums, and had employees pour the contents into 5-gallon jerry cans and take them in bucket brigades up 17 flights of stairs for 48 hours.
Peer1 CEO Fabio Banducci said the team beefed up from three the first day to an eventual 35, including customers and friends.
Sandy’s punch also hit the U.N. building along Manhattan’s East River, which was closed for three days after floodwaters breached its basement levels and is still not fully functional.
The U.N. Security Council had an urgent meeting so it gathered in a spartan, container-like structure that housed Secretary-General Ban Ki-moon’s interim offices – a building nicknamed “Walmart” and “Bantanamo” because of its utter lack of aesthetic value.
The decision to locate critical facilities in the lower levels of the more than six-decade old U.N. building had been based on analysis of weather patterns going back to the 19th century, U.N. Under-Secretary-General for Management Yukio Takasu said.
“This was a very unprecedented hurricane,” Takasu said, adding that the U.N. management might consider relocating certain infrastructure after conducting a “lessons learned” assessment of the impact of Sandy had on the United Nations.