As Stephen Harper prepares for what is shaping up to be a major remake of the federal cabinet, sources tell CBC News they expect Public Safety Minister Vic Toews will announce Friday that he is retiring from politics.

Toews’ decidedly right-wing views on criminal justice issues, and his often caustic attacks on political opponents, have made him one of the most controversial and divisive members of the Harper cabinet.

It has long been rumoured that Toews is after a federal judicial appointment, but as yet there is no indication whether the Manitoba MP will leave politics immediately for another job elsewhere, or simply announce that he won’t be running for re-election in 2015.

Either way, Toews appears destined to join a growing line of Conservative ministers expected to be demoted or dropped from cabinet altogether.

It is likely to be the biggest mid-term cabinet shuffle in Harper’s seven years as prime minister.

In the past, Harper has tended to tinker with cabinet positions to fix specific problems — for example, ditching Bev Oda for her $16 orange juice, or sidelining Maxime Bernier for his wayward ministerial briefs.

But this time, Harper’s problems are more about perception than performance, a need to paint a new face on a government otherwise appearing increasingly tired.

Defence Minister Peter MacKay is a relative newlywed and brand new dad who may well want to trade in his ministerial life of endless work and travel for more time with his family. A diplomatic appointment wouldn’t be a surprise.

If MacKay is moved out of defence, an oft-cited replacement is Heritage Minister James Moore. The B.C. MP has done a masterful job in a tough portfolio, and will almost certainly be on the move to a new position.

Women likely to figure prominently

Look for more women in high places, starting with higher-profile jobs for Labour Minister Lisa Raitt and Public Works Minister Rona Ambrose.

No matter how much shuffling the PM may do, the only way Canadians are going to perceive anything resembling a new face on the Harper administration is with some new faces around the cabinet table.

Other women to watch include:

Alberta MP Michelle Rempel, smart and a good communicator, and a virtual shoo-in for cabinet.

Dr. Kellie Leitch, a strong candidate for the health portfolio.

Manitoba MP Candice Bergen, a key figure in killing the long-gun registry and definitely ready for prime time.

Shelly Glover, also from Manitoba, a former policewoman with Métis background; has performed well as parliamentary secretary in both aboriginal affairs and finance.

The Conservatives also have good ministerial potential in a number of MPs whose cabinet aspirations have been thwarted by geography — namely, coming from a province with too many other good candidates. For instance:

Alberta MP James Rajotte, head of the finance committee and long overdue for a cabinet position.

Ontario MP Chris Alexander, only 44 and already he has been ambassador to Afghanistan and parliamentary secretary for defence.

Greg Rickford, an MP from Northern Ontario, impressive as a stand-in at Aboriginal Affairs when former minister John Duncan fell ill.

New Brunswick MP Rob Moore, who held a junior cabinet position before he was bumped to make room for former Mulroney-era cabinet minister Bernard Valcourt.

Unless Harper wants to expand the limo pool, bringing in new faces means getting rid of some old ones. That has already started.

Minister of State for Foreign Affairs Diane Ablonczy, who has long had a prickly relationship with Harper, announced Thursday she would not be running for re-election in 2015, implying she wouldn't be surprised to get dumped in the shuffle as a result.

Distancing the Senate

Senator Marjory LeBreton, who sits in the Harper cabinet as the Conservative leader in the upper chamber, also announced she is resigning that position, albeit not from the Senate. (She retires in two years.)

Not to put too fine a point on it, sources say once LeBreton is gone, Harper has no intention of replacing her in cabinet — for the first time in more than 40 years, there will be no seat in the room for the Senate leader.

Junior finance minister Ted Menzies also went public this week ahead of the shuffle, saying he had asked the prime minister to drop him from cabinet.

The Alberta MP represents some of the communities hardest hit by the recent floods, and says he would rather spend the next two years in his riding, helping his constituents rebuild their lives.

New Brunswick MP and Fisheries Minister Keith Ashfield is battling cancer, and has already asked to be relieved of his cabinet position.

In total, that would make at least five empty chairs for new faces around the cabinet table. So far.

Other names being tossed around as candidates for vacancies include:

Party Whip Gordon O’Connor. The former general may be whipping MPs into trained seals with military precision, but he is not widely loved in the Conservative caucus. If Harper wants to quell a growing revolt by frustrated backbenchers, he will likely start by getting O’Connor out of their hair.

Environment Minister Peter Kent. The Harper government is finally realizing it is not going to win pipeline and other energy fights here or in the U.S. unless it puts a more sympathetic voice on its environmental files. On Friday, Kent cryptically posted on his Facebook page that he won't be upset if he is dropped from cabinet.

Justice Minister Rob Nicholson. Once a junior minister in Kim Campbell’s government in 1993, he is now an excellent candidate for the bench.

Finally, for some of the key cabinet positions, the best move may be no move.

Atop that list is Finance Minister Jim Flaherty, long dogged by health issues and rumours he was ready to move on to a more cushy diplomatic appointment, or perhaps to the rich pastures of Bay Street.

Some insiders say it is impossible to put a new face on the Conservative government without moving arguably the second most prominent mug in the cabinet group photo.

But others argue that when the economy remains the Conservatives’ perceived strength with Canadian voters, why would Harper risk moving the finance minister?

Strong performers may stay put

It’s the same story with some other prominent ministers — they may be too good to move.

Immigration Minister Jason Kenney has proven to be a highly capable workaholic and could easily be moved to any one of a number of new cabinet positions.

Problem is, across the country he has built a virtual empire of ethnic voters for the Conservatives that Harper will be loath to risk by changing ministers.

Natural Resources Minister Joe Oliver is 72 and has had heart surgery in the past year.

But can Harper risk moving him out of the job with pivotal energy issues on the table, foremost a pending U.S. decision on the Keystone XL pipeline?

Trade Minister Ed Fast is on the verge of a free-trade deal with the European Union. Does Harper put someone new on the file?

Then there is Harper’s resident Mr. Fix It: John Baird.

By all accounts, what Baird wants, he will get — and what he seems to want most is to stay right where he is, as the minister of foreign affairs.

Of course, all the cabinet moves in the coming shuffle won’t change one salient fact:

In a government highly controlled by the Prime Minister’s Office, the one cabinet position that matters most is the one guaranteed not to change anytime soon.