An attempt to upgrade the IT system for how banks complete customer's transactions descended into chaos as Ulster Bank customers faced three weeks without access to their bank accounts.
A report into the Royal Bank of Scotland (RBS) group, which includes Natwest, by the financial regulator, the Financial Conduct Authority (FCA), detailed the cause of one of the biggest IT blunders ever to hit UK banks.
Ulster Bank customers were more greatly affected by the IT meltdown, with the majority of its systems not working until July 10 2012, three weeks after problems began on June 20.
RBS is currently two years into a three-year programme to invest an additional £750m to enhance the security and resilience of its IT systems.
The cause of colossal IT failure was the batch scheduler failure during a systems upgrade, according to the FCA.
Traditionally, through the night a bank's transactions are updated with the use of software called a batch scheduler. On June 17, the banking group upgraded this software, but once problems with its installation became apparent it tried to uninstall the upgrade, without first testing the consequences of this.
The upgraded software was incompatible with the bank's previous version, which led to a numerous incomplete transactions.
To try and correct the problem, technical support staff were manually reloading transactions. However, by the morning of June 20, while Natwest batch for the previous day had be sorted, Ulster Bank's customers transactions were behind causing a significant backlog.
This began to snowball and by June 21, Ulster Bank's transactions were more than a day behind, causing further backlogs. Problems would not be fully resolved for another three weeks.
The FCA found insufficient systems and controls to deal with potential IT risks, which included deficient testing for managing software changes, and inadequate focus on the effect of an IT incident.