When Karin Bodewits started her PhD in molecular biology at the University of Edinburgh, she felt “excited and privileged, brimming with hope”. Over the following four years, her enthusiasm waned. “That seems like another person in another life,” she recalls. “I saw a world where unreachable glory was the goal and desperation the order of the day.”

Bodewits wrote a book about her disillusionment, shedding light on a university system failing to have open conversations about the power plays between supervisors and students, the lack of mental health support, and the absence of careers advice for PhD candidates. “I wasn’t prepared well for life after my PhD,” she says. “My supervisor wouldn’t have accepted me spending time on something different than research.”

Do universities contribute enough to the UK economy? Read more

Bodewits now gives careers advice seminars to PhD students, and is disappointed to see her experience replicated. “The incentives behind PhD programmes are wrong,” she says. “At the moment it seems we’re training people because it’s cheap labour. If that’s the incentive, then for purely research-based PhDs we’re training too many for the current labour market.”

This is a common complaint from PhD students frustrated with an academic careers pipeline that requires a succession of highly competitive short-term contracts in cities all over the world. Only a small number eventually obtain a coveted permanent role; research from 2010 suggests just 3.5% of science PhDs secure permanent academic positions. There’s evidence to suggest the pressure takes a toll on the mental health of PhD students, who report a higher rate of problems than the general population.

Yet the universities minister, Chris Skidmore, recently asked universities to “think about how we can get more people staying on for PhDs in the future”. An increase in the research talent base will be vital for the government to deliver on its promised increase in investment in research and development to 2.4% by 2027, he said.

So why does the feeling that there aren’t enough jobs for PhD graduates persist? According to Clare Viney, chief executive of Vitae, which supports the career development of researchers, we will need to double the number of researchers to reach the government’s target.

But not all of those jobs will be within universities – they might be in the R&D department of a private company. “One of the challenges is that the vast majority of PhDs want to stay in academia,” says Viney. “This is where the tension comes from. Eighty per cent of researchers want to become academics but the reality is it’s a lot less [who do so]. In some disciplines it could be as low as a few per cent.”

Viney thinks universities need to have “honest conversations” about managing the expectations of PhD students. They can also better help them understand the skills they’re developing go beyond subject knowledge, such as entrepreneurship, teamwork, innovative thinking and communication. “It’s about self-awareness and emotional intelligence, focusing on researcher as much as their research,” she says.

Although people development has been a common feature of the corporate world for some time, Viney says it is comparatively new within universities. They are waking up to the fact that they’re not just responsible for “kit and keeping the lights on and making sure they have world-class facilities, but also that they’re developing people to be the best they can be”. As a result, looking after early-career researchers is a new brief within the leadership teams at some universities.

Among these is David Bogle, who is responsible for 6,000 doctoral students and 3,500 postdoctoral students at UCL. He agrees that PhD candidates can be too absorbed in the detail of their research, especially in the humanities and social sciences. “Really careful analysis of evidence for a history PhD is hugely valuable for lots of public policy work, in the media and in the law,” he says. “But they tend to still be fixated on medieval papyrology rather than seeing the broader skills.”

Bogle has looked at mental health among doctoral candidates, and developed an action plan aimed at reducing isolation and increasing support networks. “There is an issue, and I’m still not clear in my own mind what it is,” he says. “Is it because a certain type of person wants to do a PhD? Or is it because we put undue pressure on them?”

Broadly speaking, Bogle thinks a lot of work has gone into improving support, skills development and careers advice for PhD candidates in the past 10-15 years. Yet he acknowledges there is still a “communication issue with employers”, who universities have failed to convince of the value of a PhD.

“I’ve been trying to tell that story. We’ve all got to get out and do it,” he says. “These are very smart, very skilled people who are able to drive innovation in the economy.

Join us: sign up for the Guardian Universities newsletter Read more

“There’s a whole raft of smaller companies that want innovation but struggle with it. I think they’re a bit scared of how to handle these people who might challenge what they do.”

Bogle thinks the real problem is at postdoc level – the short-term research contracts that are the next step after a PhD. Although they are classed as staff (PhD candidates are students), there are far more of them in universities than there are permanent academic jobs, meaning that securing a postdoc is no guarantee of an academic career. “The mindset is wrong. It’s not enough about the people we’re developing, and it’s too much about producing papers, patents and data,” he says.

Research funders are taking action. The Wellcome Trust has recently overhauled the conditions attached to the way it funds postdocs and PhDs. According to Anne-Marie Coriat, head of UK and EU research landscape, the funder will now place “dual attention on the quality of science and the culture which sits around the way training is delivered”, including diversity, proper supervision, looking after mental health, and support for researchers to transition to the next step in their career. This includes a transition fund that can be used for any work that relates to the next step, such as writing a policy paper or undertaking a placement in a school.

'Universities stamp out creativity': are graduates ready for work? Read more

“We think this will open up the optimism with which people should be facing careers in research,” Coriat says. “These skills are required in all sorts of different settings, but what we don’t know is where those careers will be headed in 5-10 years’ time, because tech and data are really driving the pace.”

Coriat adds that quality advice on where science and research careers can lead is missing. As technology and automation transform the jobs market, she thinks gathering a “live feed” of data on how career paths are changing will be important.

Frustration with the academic career path has existed for decades, but Coriat sees the current moment as the ideal opportunity to transform research culture for the better. Meanwhile, the government’s commitment to increase research spend “links beautifully” with lots of other projects. She cautions that meaningful change “will take time to deliver”, but adds: “There’s a lot of willingness to tackle this.”