The Phoenix pay initiative that was intended to save millions by consolidating services in one place and getting rid of hundreds of in-house compensation staffers has come apart in the chaos of the federal pay crisis.

And now the Trudeau government is examining whether the model is even viable.

Government officials say the search for a Phoenix replacement will examine centralized pay operations and whether in-house compensation advisers should be permanently returned to departments to serve employees.

This week’s federal budget gave Treasury Board President Scott Brison $16 million to begin the process of replacing Phoenix, which will include whether pay operations will be centralized, decentralized or managed via a hybrid solution.

Meanwhile, Brison’s cabinet colleague, Minister of Public Services and Procurement Carla Qualtrough, says her top priority is to stabilize Phoenix and “rebuild capacity” in departments.

Public Services and Procurement Canada (PSPC) received $330 million of the $431 million the budget earmarked on Tuesday to bring Phoenix to ‘steady state.’ Most of that money will be spent by PSPC in the upcoming fiscal year.

Qualtrough told iPolitics that departments are experimenting with a re-organization of the pay centre, dividing compensation advisers into pods or teams aligned with departments.

A pilot project is underway with Veterans Affairs, Innovation, Science and Economic Development and the Federal Economic Development Agency for Southern Ontario, working with pods dedicated to them. The budget also gave departments $25 million to shore up their human resources.

“When you don’t get paid, it is a personal crisis and I don’t think anyone should have to prove hardship. …They should be able to walk down the hall and talk to someone,” said Qualtrough.

It’s been two years since the calamitous rollout of the Phoenix pay system, built by IBM using PeopleSoft software.

Today, half of Canada’s public service faces some kind of pay problem. And the 46 departments forced to lay off 1,200 pay advisers and move pay operations to a central pay centre in Miramichi, N.B. are saddled with a staggering backlog of 633,000 cases.

Politics put the pay centre in Miramichi when the Harper government decided to build it there as a political trade-off for jobs that disappeared when it closed the gun registry in the small New Brunswick town. Many say politics will keep it there.

Qualtrough said Miramichi will stay. But now, the largest federal union – which represents a growing army of compensation advisers — is pressuring the government to keep Miramichi and return compensation advisers to departments. The Public Service Alliance of Canada (PSAC) wants a permanent, full-trained cadre of advisers.

Chris Aylward, PSAC vice-president, says the original centralization plan has failed. Now he believes the government needs a ‘hybrid’ solution: keeping centralized pay operations in Miramichi, while letting departments bring back in-house pay advisers. Others argue not all departments really need compensation advisers.

Debi Daviau, president of the Professional Association of the Public Service of Canada (PIPSC), has for months pushed to scrap Phoenix and replace it with a system built by public servants. She stresses that engaging people who know the needs of the individual departments and can help employees is what’s important, not where they are located.

“It’s a virtual world, “ she said. “It is less important where they are located and what model they work under.”

PSAC says in-house compensation advisers will ensure transactions are properly entered into Phoenix on time. Most pay errors begin when information is entered late or incorrectly into human resource systems, creating a cascade of Phoenix problems.

To do that, PSPC would have to open up access to Phoenix to departments — which now have limited ‘read-only’ privileges — to allow in-house staff to enter data.

As it stands, PSAC says employees have no one to turn to with their problems, resulting in them shouldering the burden of determining whether they are paid correctly.

“For the pay system to work in the long term – no matter what the system – we will need a permanent team of well-trained compensation advisers in both the departments and at the pay centre,” said Aylward.

In terms of public perception, such a commitment would represent a major climb down for PSPC, but it’s actually been happening since the Liberals first threw the switch for Phoenix in February 2016.

In fact, the business case for the two-part $309 million pay initiative that PSPC bureaucrats drafted for the Harper government nearly decade ago has largely unraveled.

It was built on moving pay operations out of 101 departments and consolidating them at a central pay centre in Miramichi, N.B. The second phase envisioned Phoenix replacing a creaky 40-year-old pay system.

The first wave of 46 departments moved to Miramichi before rollout, but plans to move the rest have been put on hold.

The thinking behind consolidation was to automate “back office” jobs – those mundane tasks computers can do. The money saved could be better spent on big policy challenges.

But Phoenix just couldn’t fly, thanks in part to the federal pay regime’s complexity and understaffing. The system couldn’t be configured to handle the 105 collective agreements and 80,000 pay rules that many compensation advisers carried in their heads.

“Those compensation advisers were human computers and owned the intellectual property of how to calculate people’s pay. That is not a sustainable model over time,” said one technology expert. “What the government was trying to do with Phoenix made sense, but they missed on how to do it.”

The PSPC bureaucrats who wrote the business plan for the previous Conservative government figured the new system would save Canadians $690 million by 2023.

Off the bat, the government booked savings of $70 million in the first year alone. The largest share of those savings – more than 90 per cent – would come by automating the work of compensation advisers and boosting productivity so they could handle more accounts.

Advisers in departments who were managing an average of 150 accounts could, with automation, easily handle 400 accounts each, it was thought.

None of this came to pass.

When Phoenix floundered, the government changed course, hired more compensation advisers and opened satellite pay centres all within months of the rollout. There are now eight satellite pay centres.

PSPC has since hired an additional 1,400 compensation staff — more than it originally laid off. The pay centre, which was supposed to have 450 employees, grew to 700 employees. The budget’s funding is expected to shore up the number of pay advisers to 1,500. And Phoenix, which was supposed to save money, is now projected to cost more than $900 million and rising.

“While staff cuts may have been motivated by urgency in achieving planned initiative savings, in hindsight there was overconfidence in the abilities of the new pay IT solution, and a serious underestimation of the necessary role of the compensation advisers in ensuring employees are paid accurately and on time,” the Gilroy report states. “While staff cuts may have been motivated by urgency in achieving planned initiative savings, in hindsight there was overconfidence in the abilities of the new pay IT solution, and a serious underestimation of the necessary role of the compensation advisers in ensuring employees are paid accurately and on time,” the Gilroy report states.

The consolidation plan was further watered down as departments, swamped with employees who were underpaid, overpaid or not paid at all, quietly hired back pay advisers to help them sort out the mess.

The biggest testament to the value of pay advisers has come from the 54 departments which kept them. They have access to Phoenix and can enter data and, as a result, have a fraction of the pay problems compared to departments that rely on Miriamichi for pay services.

In its post-mortem, consultants Goss Gilroy said PSPC grossly underestimated the scope of the change they were bringing to the government. The project team saw compensation advisers as “lower-ranking employees” and the project as little more than “replacing a giant calculator.”

As a result, the new pay regime was managed like a giant IT project with relocation of personnel rather than as a fundamental transformation of the way pay and human resources are managed.

“While staff cuts may have been motivated by urgency in achieving planned initiative savings, in hindsight there was overconfidence in the abilities of the new pay IT solution, and a serious underestimation of the necessary role of the compensation advisers in ensuring employees are paid accurately and on time,” the Gilroy report states.

The government has hired steadily since the Phoenix crisis began: an additional 325 compensation advisers were hired in January. They’ve also offered incentive packages, retention allowances, generous overtime, extra training and flexible work arrangements.

Departments officials say they will continue to hire until the current backlog is eliminated.

The government is also setting up special units to ensure hardship cases don’t fall through the cracks and are resolved quickly. PSPC is further revamping the call centre, replacing contract employees with 100 new bureaucrats who will now have access to Phoenix and employee pay files when fielding calls.

Today, a new leadership team, headed by PSPC associate deputy minister Les Linklater, is stickhandling the stabilization of Phoenix. It is implementing an ambitious action plan to integrate pay and human resources – to try and drive the culture change that was abandoned when IBM’s planned training was dropped during the Tory era to save money.

Before Phoenix, the federal government ran its pay and human resources systems separately.

With new technology, most other large employers linked their human resource and payroll systems over the years. That didn’t happen in the public service.

The government further complicated its pay and human resource management by letting departments pick their own HR systems, leaving the government with a patchwork of 34 different human resources systems linked to a single pay system.

As one IT expert said, the compensation advisers were the “glue;” the only ones who could navigate these two separate worlds of pay and HR. (More than 200 custom-built programs were added to Phoenix to handle the interface between pay and human resources.)

Every department had its own compensation advisers and each had a roster of clients. They knew the pay rules and collective agreements affecting their departments inside and out and manually connected the separate human resource and payroll systems.

If a department promoted people or assigned them to acting positions, that information was entered twice into different systems. It went into the human resource system while compensation advisers quite independently plugged the information into the payroll system.

But Phoenix changed all that.

Managers and employees who used to have little to no involvement in their pay transactions were suddenly responsible with a self-serve system they weren’t really trained on.

And for the first time, managers, supervisors, and employees had to report their pay and HR transactions in real time.

This meant managers had to approve hours worked before the end of the two-week pay period. For years, the public service ran on reporting hours long after they worked and adjusting files and pay cheques after the fact.

But Phoenix wasn’t built for retroactivity. It works best when HR transactions are entered before they take place. Delays in entering or approving pay information resulted in a massive tangle of inaccuracies and errors.

Getting the public service to start thinking in real time about pay was a massive culture shift and required changes at all levels. Goss Gilroy found departments, especially managers, were oblivious to that fact.

When Phoenix went live, thousands of acting pay transactions, for example, that were typically claimed after the fact had to be manually processed, eating up the time of new compensation advisers and adding to the backlog of files. The same was the case with transfers, overtime, new hires, terminations and most other HR transactions.

On top of that, no one really knew what the pay procedures were. The user manual wasn’t ready and pay advisers were guessing as they went along creating more errors and backlog.

If that wasn’t enough, 700 fewer pay advisers and a neophyte pay staff at Miramichi were still learning the ropes with no system backup. It was decommissioned during rollout because there weren’t enough pay advisers left to run it.

But the catastrophic setback came with the deluge of collective agreements that required retroactive payments going back at least two years. These new contracts forced new compensation advisers to manually process pay with data retrieved from the old system they never worked on.

The old system did need replacing. It was cobbled together by decades of workarounds and temporary fixes that only the people running it really understood. The 2008 business case portrayed those experts as big a risk as the old technology. Many of the IT and pay experts were approaching retirement and their expertise and knowledge would leave with them.

A failure to replace aging technology and deal with the “instability” of retiring compensation advisers would risk “critical system failure” undermining the government’s duty to pay its employees.

The government was also having trouble finding new recruits. Young compensation advisers found the work tedious and time consuming, taking 18 months to learn the system and five years to master the job. Most were on the lookout for new jobs.

The solution to stop the turnover was to put all compensation advisers together in one spot — outside the National Capital Region — where there wouldn’t be other well-paying government jobs to go to.

That turned out to be a politically expedient solution for the Harper government when closing the gun registry. The up-front savings were also a made-in-heaven solution for the cost-cutting Conservatives.

Compensation advisers started getting pink slips in 2014. Few were interested in moving to Miramichi and started look for other jobs in government, creating a shortage of pay staff that saddled Liberals with a backlog of 80,000 cases before Phoenix even went live.

The Goss Gilroy report, which has become the ‘how-to’ guide for large projects, offered this advice among its recommendations for the next incarnation of Phoenix .

“The experience … highlights the imperative of considering and planning for sufficient numbers and skilled capacity to support change processes, in particular when the change is major and complex and relies on subject matter experts, such as compensation advisers, and a high number of users, including managers and employees.”