Hewlett Packard Enterprise (HPE) has struck a settlement with the Australian Taxation Office (ATO) over the “unprecedented” failure of its storage hardware in late 2016.
Australian Commissioner of Taxation, Chris Jordan, told Senate Estimates committee members on 30 May that the ATO had reached a commercial settlement with HPE, the detailed terms of which are subject to contractual confidentiality.
“The settlement recoups key costs incurred by the ATO, and provides additional and higher grade IT equipment giving the ATO a world-class storage network,” Jordan said.
The move to strike a settlement agreement following the hardware failure and subsequent systems outage echoes the settlement IBM reached with Australian Government over the troubled eCensus project with the Australian Bureau of Statistics (ABS) last year.
Jordan also revealed the preliminary findings of various investigations into the storage hardware failure last year, citing a combination of factors.
According to Jordan, preliminary analysis from HPE, PricewaterhouseCoopers (PwC) and the ATO itself revealed a number of problems, including that the fibre optic cables feeding the troubled HPE storage area network (SAN) were not optimally fitted.
Additionally, disk drives on the HPE 3PAR SAN that failed had software bugs that made the stored data on the drives inaccessible or unable to be read and some monitoring features were not activated, including a “back-to-base” tool to report operating errors.
Jordan revealed that a report on the outage will be published within the next week, and will provide the findings of all the reviews to date from the three entities carrying out the investigations.
Jordan said that, in summary the report will indicate that the turnkey service of data storage as per the 3PAR SAN provided by HPE "failed" the ATO.
“The SAN design and configuration meant we had an over emphasis on performance features rather than stability or resilience - a relatively small disk drive failure had a large impact - only 12 of some 800 disk drives failed, but they impacted most ATO systems,” Jordan said.
“The recovery was slower because some of the recovery tools required were themselves stored on the same SAN that failed,” he said.
At the same time, Jordan said that the ATO’s business continuity mechanisms, communication and engagement worked well, but into the future would need to be more inclusive of the ATO’s partners, such as software providers, tax professionals, and the superannuation industry.
“I want to assure the community that our systems are restored, no taxpayer data has been lost and we are tracking well for delivery of Tax Time 2017,” Jordan said. “We’ve already rebuilt our storage system to the latest in world-class standards, replacing the faulty equipment with new technology and we’re reviewing our IT framework to enhance stability and resilience more generally.”
The hardware trouble struck the ATO in December last year when an “unprecedented” failure in December last year of 3PAR storage hardware that had been upgraded in November 2015 by HPE resulted in widespread outages among many of the ATO’s systems.
In early February, Jordan took aim at HPE, suggesting that the agency’s technology partner had “failed” to reliably provide it with its contracted services.
“Initial indications are there has been a failure by Hewlett Packard Enterprise (HPE) to provide contracted services in a reliable way and ensure stability of our systems,” Jordan said in a statement published on 8 February.
