A bunch of ‘he said, she said’ has erupted as IBM details where it all went wrong with Australia’s 2016 Census.
August 9 was census night for Australia. Unlike other years, this year was pegged as the year Australia would transition from the paper census to an online form. IBM was put in charge of the transition, spending almost $10 million on the online census solution.
Prior to the census, much of Australia was preoccupied with concerns about the security of their information, while a smaller group expressed doubts about the capacity of the census website to accommodate millions of Australians logging on at once.
The latter turned out to be the bigger concern, with the census website being inaccessible for 40 consecutive hours due to a series of DDoS attacks. Following this, there was an inquiry into the outage, with IBM being forced to explain the failure to a Senate committee yesterday.
Router issues
According to IBM senior engineer Michael Shallcross, two routers were being used to channel data traffic to the website from its two ISP partners. One of these routers failed after it was rebooted following a DDoS attack, and was left inoperable for over an hour.
“We did, during the lead up to the census, test the impact of a failure of that router and test that the failover mechanisms on the rest of the site worked effectively,” Shallcross told the committee.
“But we tested that router failure by simulating it, which is relatively easy to do in a repeatable fashion. If we had our time again, we would probably test a hard ‘power-off, power-on’ that router. That would have discovered earlier that we had that reboot and configuration and loading problem.”
Liberal senator June Hume was one of many critics who suggested that IBM was not rigorous enough in its testing.
“That’s the sort of level of technical competence I have with my computer,” she told the committee.
Incompetent telco providers
IBM has criticised telecommunications provider Nextgen and its upstream partner Vocus in the implementation of its geo-blocking DDoS prevention plan, dubbed ‘Island Australia’.
“It’s apparent from the submissions brought by Nextgen and Vocus that perhaps the internal communications had not conveyed adequately the intent and instructions of and surrounding the implementation of Island Australia,” said Shallcross.
“We, as the prime contractor, dealt with both Telstra and Nextgen as our ISPs, and expected them, as large internet service providers, to be able to implement those instructions correctly.
“Telstra did [this] for us in 2011, and again in 2016. We had difficulty with the implementation by Vocus of those particular instructions.”
Nextgen has previously defended itself in a submission to the committee, stating that it “provided all possible assistance” to IBM in the implementation of Island Australia.
The company also noted that it had suggested an alternate DDoS protection option, however it was rejected by IBM.
IBM has claimed that this suggestion was unsuitable.
Foreign attacks
It has been determined that the DDoS attack that eventually overwhelmed IBM’s infrastructure came from Singapore.
As mentioned above, IBM was utilising a geo-blocking strategy, which the company felt was appropriate as the census website was only meant to be used by people who were in Australia at the time.
Special advisor to the Prime Minister on cyber security, Alastair MacGibbon, has expressed his disagreement with the strategy.
“There certainly were better alternatives,” he told the committee.
MacGibbon noted that while the strategy was based on preventing overseas traffic, it is not uncommon for internet traffic from Australia to appear as if it is from overseas, and that the password reset system for the census website itself relied on data coming from overseas.
“There was a fundamental failure in the logic of an ‘Island Australia’. I could see it as part of a series of protection, adding some value. But to solely rely on it, clearly, was a failure,” he said.
There have also been claims that the strategy did not receive approval from all necessary parties, namely the Australian Signals Directorate (ASD), which provides cyber security for the government.
“It was discussed with ASD, but I’m not aware that the ASD ever passed a comment saying one way or another that they endorsed it or not,” Shallcross told the committee.
The aftermath
Managing director of IBM Australia, Kerry Purcell, has apologised for the handling of the census website, but has made it clear that no one has been disciplined or fired over the failure. He stated that negotiations were underway in regard to compensation costs, but neglected to reveal an estimate of the figure.
ABS told Senate Estimates last week that the outage would add an extra $30 million to the cost of the census.
ABS chief David Kalisch has apologised for the outage, promising that the 2021 census will take on a “more rigorous approach”.