One of Stephen Harper's most trusted political lieutenants is calling on Rob Ford to vacate the Toronto mayor's job, saying the municipal leader has "brought dishonour" to public office.

Employment Minister Jason Kenney is the first Harper government cabinet minister to publicly reproach Mr. Ford, who as mayor has been a valued ally of the federal Conservative Party.

The Calgary minister, who successfully steered immigration reforms and wooed new Canadians to the Tories, is among a select group of cabinet members with the authority and autonomy to speak their mind. Mr. Kenney is also seen as a strong possible leadership contender.

Story continues below advertisement

"I will say, though, as an elected official, that I think Mr. Ford has brought dishonour to public office, the office of mayor and his city," Mr. Kenney said, characterizing his statement as his personal opinion. "I wished he had taken a leave of absence some time ago to go and deal with his personal problems. But not having done that, I personally think he should step aside and stop dragging the City of Toronto through this – through this terrible embarrassment."

Many Conservative MPs have avoided criticizing Mr. Ford since his admission earlier this month that he had smoked crack cocaine and instead urged the mayor to seek help. They do not want to anger his political base, known as Ford Nation, which potentially overlaps their own support.

But reticence is being replaced with a growing sense that the mayor is a liability. On Monday, Prime Minister Stephen Harper's office signalled a change when it called the most recent "allegations" against the Toronto mayor "troubling."

Finance Minister Jim Flaherty, who broke down earlier this month when asked what Mr. Ford should do, repeated on Tuesday that the mayor "needs to decide for himself," but that he remains a friend. "Yes, of course, he's my friend. You don't have a friend one day, and not a friend the next day. What kind of a person is that?"

Mr. Ford was away from City Hall for part of Tuesday making house calls and continuing his circuit of television interviews while the first signs of his shrinking sway with Toronto's government were on display: workers changed locks and staff packed boxes.

He also lost his highly touted TV talk show, in which he starred with his brother Doug, when the conservative cable news channel Sun News axed Ford Nation less than 24 hours after its debut on Monday night. Despite record ratings for the network, a Sun News executive said it was too expensive and difficult to produce.

In his absence from City Hall, Mr. Ford's senior advisers and deputy mayor Norm Kelly divvied up the mayor's employees to fit his slimmed down job and office budget – slashed by 60 per cent on Monday in an unprecedented council vote. Mr. Ford's office was reduced to nine from 20, with top officials – his chief of staff, deputy chief of staff and director of policy – all opting to work for the newly empowered deputy mayor's office.

Story continues below advertisement

"It's not [the mayor's office.] Not any longer," Mr. Kelly said of the glass-walled area where Mr. Ford's staff sit.

Mr. Ford will keep his office, with its view of the square in front of City Hall, along with its adjoining staff and reception areas. The staff who stay with him will be located there.

Mr. Kelly confirmed staff were unable to inform the mayor about the changes before they were made public. "I think efforts have been made to reach him, but he's been a moving target today," he said.

As reporters grilled the deputy mayor on the changes, Mr. Ford walked off the elevator behind them, but only a few reporters broke away to chase him, a stark contrast from the media frenzy his presence generated just a day ago. After the deputy mayor spoke, even fewer stuck around to wait for the mayor. As the dinner hour approached, Mr. Ford remained in his office and was making counter-offers to keep some staff. He later left without speaking to reporters.

Tuesday's staffing shuffles will put all the policy and council relations duties in the office of the deputy mayor, who will also chair the executive committee and take over many of the mayor's duties.

Toronto city council voted 37-5 on Monday to curb the mayor's power, an unprecedented measure aimed at bringing stability to city hall after months of controversy generated by Mr. Ford's behaviour.

Story continues below advertisement

Earl Provost, now the chief of staff for the newly expanded office of the deputy mayor, said the mayor's budget – set at $712,000 for the remainder of his term in 2014 – dictated the decisions. "This is a direct result of what happened yesterday. It's the reality of his budget being slashed," he said.

The decision on how staff would be divided was based on job descriptions, skills, staff preferences and the dollars available to the mayor, Mr. Provost said. "Everyone feels bad about this, but this is the reality of the new situation," he said.

Many of the mayor's staff had privately expressed mounting unhappiness with his antics. Mr. Provost, who Mr. Kelly said asked to move to the deputy mayor's office, was ready to resign after Mr. Ford's lewd outburst last Thursday, sources in the mayor's office said.

"We very much look forward to working with the deputy mayor and continuing the agenda of fiscal responsibility," said Sheila Paxton, who will move over to be director of policy for the deputy mayor.

A city hall source said last night that Mr. Ford's manager of scheduling, Dan Jacobs, will take the senior post in his office. Another one of the nine who are staying with Mr. Ford is Amin Massoudi, the mayor's special assistant for communications. It is not clear if Dave Price, a long-time friend of the mayor's brother who a Globe and Mail investigation identified as someone who dealt hashish with Doug Ford in the 1980s, will be offered a post.

Other staff staying with the mayor include his recently hired driver Jerry Agyemang, Mayor Ford's long-time staff member Tom Beyer, event manager Victoria Colussi, and special assistants Jonathan Kent and Christine Maydossian. It was not clear if another, J.C. Hasko, stayed with the mayor or moves to the deputy mayor's office.

Story continues below advertisement

Mr. Kelly said he is looking forward to working with his new staff, who chose to move to his office. "I've got a young staff that are eager to meet the challenges that lay ahead," he said, adding the locks and security passes were changed in the area where the mayor's former staff work so Mr. Ford cannot access it without permission from the deputy mayor.

In an interview on CP24 earlier in the afternoon, Mr. Ford expressed a sense of betrayal that Mr. Kelly voted to cut his budget. "He said he was going to support me, and at the end of the day, his record speaks for itself. He didn't support me," he said.

The deputy mayor said he was not bothered by Mr. Ford's comments.

"Politics is war without bullets," he said. "There's a loyalty that transcends the office and the person and that's a loyalty towards the residents of Toronto and their expectation of good government."

With reports from Simon Houpt and Ann Hui in Toronto and Kim Mackrael and Campbell Clark in Ottawa. Kaleigh Rogers is a freelance journalist.