One of Neil Woodford’s biggest City backers dropped the fund manager on Wednesday, hours after he apologised to investors on YouTube for suspending trading in his largest fund.

Wealth manager St James’s Place, announced it had terminated the stock picker’s contract to manage three of its funds – worth a total of £3.5bn. That mandate, which Woodford has held since launching his own firm five years ago, has now been handed to rivals Columbia Threadneedle Asset Management and RWC Partners.

It is the latest blow for the UK’s most renowned fund manager, who said on Wednesday morning he was “extremely sorry” for taking the decision to block investors from pulling cash from his flagship £3.7bn Equity Income Fund. However, he insisted it was in their best interests.

In a video filmed in his offices in Oxford and posted on YouTube, Woodford said: “As difficult a decision as this is, and clearly frustrating for you, our investors, we felt this was necessary to protect your interests.

Profile Who is Neil Woodford? Show Hide Neil Woodford was once the UK’s biggest star fund manager, personally managing a £25bn mountain of money on behalf of pension funds and other investors at Invesco Perpetual. When he decided to quit Invesco and go it alone in 2013 it was a huge shock for the fund management industry. Invesco shares slumped by 7% on the day he announced his departure. At Invesco Woodford held control of huge stakes in some of the UK’s biggest firms, and his opinions mattered. His criticism of AstraZeneca chief executive David Brennan in the 2012 shareholder spring was widely regarded to have cost him his job, and his critique of BAE’s attempted £28bn merger with Airbus is acknowledged as one of the reasons the deal collapsed. Woodford, who was widely referred to in the media as an investment “hero” and fund management “star”, had done exceedingly well over his quarter century there. A £1,000 investment placed when he started at the firm in 1988 would have risen to £23,000 by the time he left. Woodford accidentally fell into fund management and hadn’t heard of the term until he rocked up in the City in the 1980s sleeping on his brother’s floor while looking for a job. He got his first break in insurance, before drifting into fund management. He had left school wanting to fly fighter jets but couldn’t pass the RAF’s aptitude test, and instead read economics and agricultural economics at the University of Exeter. Feeling he had outgrown Invesco Perpetual, he set up his own firm Woodford Investment Management in 2014, on an industrial estate near Oxford. Within two weeks of launching, he had raised £1.6bn, a UK record, and it quickly grew to £16bn. In its first full year his flagship fund returned 16% and Woodford, a devotee of veteran US investor Warren Buffett, was dubbed the “Oracle of Oxford”. Asked if he ever doubted his judgment, Woodford once said: “Daily. You must never, as a fund manager, stick your head in the sand saying ‘everybody go away, I’m right, I’m right, I’m right’. You’ve always got to expose yourself to criticism and the analysis that you may be wrong.” Woodford went on to say that the secret of successful fund management was a balance of arrogance and humility. “You have to have a sufficiently strong arrogant gene to back your judgment, back your conviction. If you didn’t, you would end up with a portfolio that looks very much like the index. But, equally, you must have the humility to accept that you will get things wrong.” Rupert Neate Photograph: Jenny Goodall/Rex Features

“I’m extremely sorry that we’ve had to take this decision. We understand our investors’ frustration. All I can say in response to that is that this decision was motivated by your interests.”

Woodford's fall shows that investment genius rarely last | Nils Pratley Read more

Woodford said he would keep investors informed and the fund would be reopened when “appropriate”, but did not comment on when that might be.

The fund will be suspended for at least four weeks, during which time investors cannot pull money out or put money in. The decision to freeze the fund was taken after a rush of customer withdrawals following a series of bad market bets.

Investment group Hargreaves Lansdown, whose clients last year accounted for more than 30% of Woodford’s now blocked fund, said on Wednesday it would waive its fee for affected clients as long as the fund was suspended.

Emma Wall, head of investment analysis at Hargreaves Lansdown, said: “We have been in communication with Woodford Investment Management to explain why we think this is the right thing to do and have put pressure on them to do the same.”

Woodford investors are believed to be still liable for fees of up to 1.7% despite the suspension.

Andrew Bailey, chief executive of City regulator the Financial Conduct Authority, defended Woodford’s decision to suspend the fund.

“The alternative would have been much more disorderly,” Bailey told Bloomberg TV in his first comments since the gates were closed on investors on Wednesday.

He assured the FCA would be watching Woodford closely as he starts to sell-off his stakes in privately held companies, which are more difficult to turn into cash. Bailey said it was important for funds like Woodford’s to invest in unlisted firms.

“There is always a balance here, we have to have an economy in which there’s investment in illiquid assets. And then it’s a question of how we do it to preserve the integrity of markets.”

Bailey cautioned that large funds need to be monitored. “Ten years ago we were dealing with the problem of banks that were too big to fail, of course we don’t want funds that are too big to fail.”

However the FCA said it was weighing a potential investigation into Woodford, linked to the listing of some of the fund’s private company investments in Guernsey. It was meant to help get around rules that limit the proportion of unlisted investments at 10%, while the total size of his flagship fund continued to shrink.

“Where the FCA believes there are circumstances suggesting serious misconduct or non-compliance with the rules it may open an investigation.”

A request by Kent county council’s pension fund to withdraw its £263m investment is thought to have prompted Woodford to take the decision to suspend the fund.

Addressing investors, Woodford said in the video that putting a temporary halt to withdrawals would afford the fund the “time and space” needed to reduce exposure to illiquid stocks and private companies.

He said the capital would be redeployed into “more liquid stocks in the FTSE 350 but primarily in the FTSE 100 that fit in with my core strategy, but which are more liquid and more conventional, if you like”.

Comments posted underneath the video were mixed. Ian Jenkins wrote: “Caught out by his own hubris. Overreached himself.”

Meanwhile, Philip Fox wrote: “You tried to instigate a terrible takeover of Provident Financial. Of which I own shares to the benefit of you and a few others. Your strategy was terrible but you were not interested in people or companies, just short-term share rise.”

However, others were more supportive. “Neil is a good guy, I put my kids through uni thanks to him and his funds. I’m keeping the faith, no knee jerking here, in for the long term, £10k invested in 1990 now £280k. Not bad, cheers Neil,” wrote one supporter under the handle “Ham head”.

Keith Marsh wrote: “If anyone can turn this around, it is that man. Investors, relax and let him do what he does and has consistently done best.”

Assets in the Woodford Equity Income Fund fell by £560m in May to £3.7bn, according to data compiled by Morningstar, a financial research company.

Sign up to the daily Business Today email or follow Guardian Business on Twitter at @BusinessDesk

Woodford built his reputation over 26 years at Invesco Perpetual, where he personally managed £25bn on behalf of pension funds and other investors. He left Invesco in 2013 to set up on his own.

Woodford paid himself and his business partner, Craig Newman, nearly £37m in the 2017-18 financial year.

The decision to suspend his flagship fund had a knock-on impact , sending shares in his Patient Capital Trust fund down nearly 7%, having tumbled 20% a day earlier.