The largest union representing federal government workers is formally requesting a federal public inquiry into what went wrong with the now $1 billion Phoenix pay project in the wake of a scathing report by Canada's auditor general.

"By not listening to the people doing the work, or the unions that represent them, the government created a recipe for disaster," said Chris Aylward, national president of the Public Service Alliance of Canada. "We also need to build on what we found out today through a national public inquiry so that we can ensure that nothing like this ever happens again."

Since the federal government's Phoenix pay system was launched in 2016 thousands of public servants have been improperly paid.

Canada's Auditor General Michael Ferguson called the Phoenix pay system "an incomprehensible failure" in a report released Tuesday. Canada's largest union representing federal government workers has called for a public inquiry. ( The Canadian Press/Sean Kilpatrick)

Earlier Tuesday, Canada's Auditor General Michael Ferguson described Phoenix as "an incomprehensible failure" which he laid mainly at the feet of three so-called "Phoenix executives," high level bureaucrats in charge of implementing the new pay system.

"The decision by Phoenix executives to implement Phoenix was unreasonable according to the information available at the time," said Ferguson in delivering his spring report.

"As a result, Phoenix has not met user needs, has cost the federal government hundreds of millions of dollars, and has financially affected tens of thousands of its employees."

Top bureaucrats a target of auditor general's criticism

Ferguson noted there was plenty of blame to go around, from the previous Conservative government that approved the Phoenix project to the current Liberal government that launched the system.

However, he said a great deal of power rested with three top bureaucrats who oversaw Phoenix, who in some cases kept key information from the department's deputy minister, and who also ignored dire warnings from outside departments and consultants.

"The project executives were to blame for the project failures. The deputy minister who was in place when the system was launched is accountable for the failure that happened on his watch," Ferguson said.

"The former government is accountable for not having built an appropriate oversight mechanism…and the current government is also responsible for fixing the problem, a fix that will have an incremental cost of more than $1 billion and will take years."

Chris Aylward, vice president of the Public Service Alliance of Canada, is calling for a national public inquiry into Phoenix. (Matthew Kupfer/CBC)

"The culture has to change"

The "obedient" culture of the public service also contributed to the failure of Phoenix, Ferguson said, as public servants reduced budgets in an effort to please political masters who like to see projects completed on time and on budget.

With large, transformative IT projects, Ferguson said, that approach risks "a failed transition and higher, rather than lower, costs in the long term.".

"I don't have a set of instructions to fix a broken government culture," said Ferguson. "The culture problem is real and it urgently needs to be fixed."

Poor management by Public Services and Procurement Canada

The audit also found that Public Services and Procurement Canada, the department that implemented Phoenix, failed to properly manage the pay program.

It found the department did not fully test the system and had no plans to upgrade its software in the future.

In order to meet strict budgets and time constraints, the department also made unreasonable cuts, according to the report.

Phoenix executives were more focused on meeting the project budget and timeline than on what the system needed to do. - Auditor Genera Michael Ferguson

For instance, Treasury Board approved spending of only $155 million even after IBM, the company contracted to modify the software, had told the department it would cost $274 million to build and implement Phoenix.

The audit also found that Phoenix executives worked with IBM to find ways to reduce the scope of work to fit the approved budget.

That meant the testing of some pay processes was eliminated, the project schedule was tightened, the number of IBM and PSPC employees devoted to Phoenix was reduced, and more than 100 pay processing functions — such as those for managing requests for retroactive pay and acting pay — were removed from the program, with plans to add them later.

Carla Qualtrough, Minister of Public Services and Procurement, said the government accepts all of the auditor general's recommendations on Phoenix and is already working to implement them. (Sean Kilpatrick/Canadian Press)

As a result, when Phoenix went live, "it could not perform some critical pay functions," states the audit. "In our opinion, these weaknesses were serious enough that the system should not have been implemented."

The audit also found Phoenix went ahead despite security risks known to management, and noted "the Privacy Commissioner of Canada has reported numerous privacy breaches of federal employees' information in Phoenix, after it was in place."

Pay centre in Miramichi overwhelmed

The plan to centralize the pay centre in Miramichi, N.B. was also motivated by a desire to save money.

The audit points out that under the government's former pay system, each compensation worker handled an average of 184 employee pay files. It was anticipated the updated technology of the new Phoenix system would allow them to handle 400 files each, more than double their previous workload.

However, by July 2015, PSPC managers realized that pay advisors in Miramichi could not handle this large and growing number of files.

"Phoenix executives did not ensure that pay advisors could handle the files already assigned to them before doubling their workload," states the audit.

Auditor General Michael Ferguson released his spring audit Tuesday in Ottawa 1:08

Important information withheld from deputy minister

Just before Phoenix launched in February 2016, the deputy minister of PSPC met with the three top executives in charge of it, but was not informed by them of the significant problems that had been reported.

"The deputy minister did not receive independent advice on the Phoenix pay system's readiness and relied solely on Phoenix executives," states the report. "Phoenix executives, in effect, decided to implement Phoenix."

'We didn't create this problem, but it is ours to fix'

Minister of Public Services and Procurement Carla Qualtrough responded to the auditor general's report with a statement typical of the government's messages about Phoenix over the past two years.

"It bears repeating, we didn't create this problem, but it is ours to fix," said Qualtrough, adding that budgetary constraints imposed by the previous Conservative government along with a "corporate culture which created a fear of mistakes" led to the Phoenix disaster.

The government accepts all the auditor general's recommendations on the Phoenix pay system and has already taken steps to implement them, Qualtrough said in the statement.