Recent scandals in the Department of Veterans Affairs and the Internal Revenue Service demonstrated that it’s almost impossible to fire federal employees, many of whom reportedly intend to go rogue by not implementing President Donald Trump’s agenda.

“It’s hard to argue we have an accountable government when someone can’t be fired for years at a time,” @bgwilterdink says.

Conservatives are hopeful the time has come for civil service reform that would rein in this permanent class of government workers who have voiced outright hostility to the new administration. Some have even called it the “fourth branch of government” or “alt-government.”

“This is a situation where people voted and elected a president who is lawfully trying to complete those tasks [he promised in the campaign], while unelected bureaucrats are willing to overturn the will of the people,” Ben Wilterdink, director of the American Legislative Exchange Council’s (ALEC) Task Force on Commerce, Insurance and Economic Development, told The Daily Signal.

Among federal employees, about 95 percent of political contributions went to Democrat Hillary Clinton during the presidential race, according to an analysis by The Hill.

Some of those federal workers are now in consultation with departed Obama administration officials to determine how they can push back against the Trump administration’s agenda, The Washington Post reported last week.

At the State Department, for example, nearly 1,000 government workers signed a letter protesting Trump’s executive order on refugees. A few days later, Trump had to fire acting Attorney General Sally Yates after she announced she wouldn’t defend the administration’s refugee policy.

White House press secretary Sean Spicer said State Department employees who oppose the policy “should either get with the program, or they can go.”

“If a federal employee doesn’t like the ideological foundation or likely outcomes of a presidential directive, it doesn’t mean that the directive is not legal. It means that the views of the federal employee are in conflict with the views of the president who runs the federal government,” said Neil Siefring, vice president of Hilltop Advocacy and a former Republican House staffer, in a column for The Daily Caller.

“In that instance,” Siefring added, “the solution should not be to resist the actions of the president in their professional capacity as a career civil servant in the workplace. The solution is for that federal employee to honorably resign, not actively or passively hamper the White House.”

What if an employee won’t resign? Addressing the problem with the federal workforce won’t be easy, according to experts interviewed by The Daily Signal.

“You can fire federal employees, it’s just that nobody wants to put up with the process,” Don Devine, former director of the Office of Personnel Management during the Reagan administration, told The Daily Signal.

Multiple appeals can be made through the U.S. Merit Systems Protection Board, the Equal Employment Opportunity Commission (EEOC), and the National Labor Relations Board.

“It’s almost impossible to discipline employees because it can be appealed to through the merit system, the labor relations systems, or through the EEOC,” Devine said. “We don’t have a civil service system; we have a dual civil service-labor relations system.”

During the Obama administration, two of its biggest scandals involved the IRS and Department of Veterans Affairs. In 2013, a Treasury Department inspector general report determined the IRS had been targeting conservative groups. In 2014, a VA inspector general’s report revealed falsified appointments in which some veterans died while waiting for care.

Years later, conservatives remain frustrated that federal workers weren’t held accountable.

“I will take your IRS employees and raise you the EPA, where story after story, a worker was viewing porn on work time and couldn’t be fired because the process is fraught with appeals,” Wilterdink said. “It’s hard to argue we have an accountable government when someone can’t be fired for years at a time.”

Earlier this year, the U.S. House revived the Holman Rule, named after a Democrat congressman who introduced it in 1876. It would allow lawmakers to cut the pay of individual federal workers or a government program.

There are other proposals for holding federal workers accountable. House Oversight and Government Reform Chairman Jason Chaffetz, R-Utah, introduced a bill in January to hold seriously tax delinquent people ineligible for federal civilian employment, federal contracts, or government grants. This bill was proposed in response to IRS data that found more than 100,000 federal civilian employees owed more than $1 billion in unpaid taxes at the end of fiscal year 2015.

Adding to the challenge is the process commonly known as burrowing. Frequently, political appointees from one administration convert to a career position that comes with civil service protections, allowing them to continue implementing policy—or resisting the new administration’s approach.

The Pendleton Civil Service Reform Act of 1883 was passed to stop raw political party appointments from securing federal government jobs, or a spoils system. The law introduced the merit system into hiring practices and made numerous civil service positions untouchable after they were filled.

However, burrowing has caused a de facto spoils system, Wilterdink said, because, “the pendulum has swung so far to protecting federal employees” that it allows administrations to keep their people in office long term.

Significant reform doesn’t mean recreating a spoils system, according to Robert Moffit, a senior fellow at The Heritage Foundation who was an assistant Office of Personnel Management director during the Reagan administration. Moffit said a balanced approach would be more desirable.

“You need to have strong managers in each agency to make sure the president’s agenda is properly executed,” Moffit told The Daily Signal. “You must also have a bright line between career and non-career staff so there is no politicization of the merit system.”

Moffit also supports legislation to allow the president to order the firing of career officials who either “broke the law or severely undermined the public’s trust.”

“Even President [Barack] Obama referred to what IRS officials did as outrageous and nothing happened,” Moffit said. “The VA matter is still unresolved. The people responsible for those waiting lists aren’t accountable and people died.”