High-pitched sound destroys bank’s data storage drives. Problems during a planned fire drill shut down ING Romania’s entire data center for ten hours on Saturday, in an event that looked like a MythBusters episode.
The fire drill involved the release of Inergen gas, a type of inert gas used to extinguish fires in data centers without damaging equipment.
Not everything went was planned during the release of the Inergen gas, which resulted in abnormal and high-pitched sound levels that resonated and affected the bank’s data storage hard drives, causing them problems writing and reading data from their disks.
Everything was down. Everything!
This resulted in the damaging of some equipment and the immediate shutdown of the entire bank’s suite of services. At around 13:00 local time, the bank’s entire ATM infrastructure stopped functioning, along with its PoS services, online payment system, Homebanking service, commercial website, and even internal email and ticketing support system.
This left the bank in the awkward position of not being able to communicate with its clients for a few hours.
The first to figure something was wrong were Softpedia’s Romanian forum users. Attempts from Softpedia to contact the bank at that time were unanswered.
This was not an ATM heist or SWIFT attack, even if it looked like one
Due to the wide-reaching and devastating blackout of its IT system, it took the bank over four hours to post its first message on its Facebook account. An ING spokesperson told Softpedia at the time that this was “a general technical issue.”
Users complained on Facebook, Twitter, and forums, and speculation about cyber-attacks was rampant. The bank’s website, which often came back online only to succumb a few minutes later, presented the classic signs of a DDoS attack.
The shutdown of its entire ATM and transactions service on a weekend day also made some speculate that a SWIFT attack or a coordinated ATM heist had taken place. These latter attacks often occur during weekends, when banks aren’t fully staffed.
Safety and security procedures delayed the data restoration process
Bank services started coming back online after 20:00 local time and were fully restored by 23:00 the same day. On Sunday morning, the bank issued a statement about what happened during the previous day.
ING Romania explained that while it would have been easy to restore from a backup and start services immediately, due to the highly sensitive data found on its data center’s storage servers, they had to follow a strict set of procedures and tests before putting each of its services online.
The bank is currently undergoing a diagnostics and analysis phase of the entire incident. ING Romania also apologized for its failure to inform customers of the issues earlier, but said this would have been practically impossible since it couldn’t reach its customer database at that point in time.
Bank officials say they’ll reimburse costs to all ING clients who used ATMs of other banks to withdraw money on Saturday.
Softpedia collaborated on this story with Andrada Fiscutean. You can read more about the science behind the data center’s failure in her Motherboard feature. Below is a video released by the bank’s PR staff from Saturday’s mini-crisis.