Local authorities are having to spend millions of pounds on social work agencies as they struggle to recruit permanent staff, with some authorities employing nearly half of their children’s social workers through private companies, a Guardian investigation has found.

Data obtained through freedom of information requests shows that many English councils are routinely spending tens of millions of pounds – a total of at least £335m in 2017/18 – hiring agency social workers.

Experts said the difficulty experienced by councils in attracting permanent staff meant vulnerable children and families were often seeing multiple social workers in a single year, making it harder for them to engage with services.

They said the large-scale use of agency social workers was a poor use of dwindling local authority funds, as locums received a higher hourly rate than permanent staff, on top of the fee paid to the company they were employed through.

As of September last year, 26 local authorities got more than 30% of their children’s social work staff from agencies. Forty-eight per cent of Swindon council’s children’s social workers were employed through an agency, as were 47% of Croydon’s.

Tameside and Lambeth recruited 43% and 39% through agencies respectively. This compares with an average among English councils of 15%. Only 10 out of 152 local authorities in England hired one or no children’s social workers from agencies in September last year.

Ray Jones, a former director of children’s services and an emeritus professor at Kingston University, said working conditions for social workers had deteriorated, with bigger caseloads, less support and inadequate pay, which made locum work more attractive.

“You’re paying more for a poorer service [with agency workers] because what you need in terms of children’s and adults social services is continuity – people who know the people they are working with, can build relationships with those families over time, and know their history,” he said.

The Guardian reported in January that nine in 10 local authorities spent more than what they had budgeted for on children’s services last year, as the number of children being taken into care hit at a 10-year high.

According to data from 129 out of 152 local authorities across the country, councils spent a total of £335m on agency social workers in 2017/18. This represented a slight fall from £360m in 2016/17, which in turn had risen from £342m in 2015/16.

The biggest spender in 2017/18 was Northamptonshire, which declared itself effectively bankrupt in February last year. It spent £12m, which represented a drop from £18.4m the previous year. A council spokesperson said reliance on agency staff was an issue faced by nearly all local authorities across the country, and that it had managed to reduce its use of agency workers by 23% since 2017.

Jones – whose book In Whose Interest?, on the privatisation of child protection and social work, was published in December – said that among the reasons for the rise in the use of agency workers were government cuts to local authorities, which had made social work a less attractive profession.

“More people are getting into difficulty because of increasing poverty, so there’s more pressure on services,” he said. “At the same time there have been funding cuts that limit what you can do to help.”

Q&A Tell us about your experiences Show Hide We would like to hear from readers who are agency social workers. You can share your experiences with us by filling in this encrypted form – anonymously if you wish. We will not publish anything without contacting you and only the Guardian has access to contributions. You can read terms of service here.

Government figures show there were 5,810 children’s social worker vacancies in England in September last year, with rates ranging from 6% in Yorkshire and Humber to 26% in London. Seventy-eight per cent of agency children and family social workers working for local authorities in September last year were covering vacancies.

In response to the increasing use of agency social workers, local authorities across the UK have worked together to cap the amount they pay. However, a report by the website Community Care found that social workers hired through private companies could still earn thousands of pounds a year more.

One of the biggest providers of agency social workers is the UK-registered Liquid Personnel Limited, whose turnover increased to £101m in the 2017 calendar year. Its ultimate parent undertaking is Indigo Cayman Limited Partnership, which is registered in the Cayman Islands.

Rachel Wilson, a resourcing manager at Swindon council, said the number of agency workers it hired had crept up over the past 12 to 18 months. In May, Ofsted found “significant weaknesses” in the local authority’s children’s services.

“There has just been a lot of competition for that skillset,” said Wilson. “We’ve got a lot of neighbouring local authorities who are all fishing in the same pool.” The council has launched a series of initiatives with the aim of attracting and retaining social workers, including launching a social work academy and partnering with schools and universities.

Mark Green, the council’s social care service manager, said one of the primary downsides of hiring high numbers of locum workers was a lack of consistency for service users. “If you’ve got a churn and perhaps have four or five social workers in a year, it’s really not very good if you’re a looked-after child or a family on a child protection plan,” he said.

Lambeth council said it was developing opportunities for career progression and focusing on workforce wellbeing in an effort to reduce its use of agency social workers. Croydon said it had recently recruited an extra 36 social workers to reduce caseloads, giving social workers more time with children and families and boosting morale.

Tameside council said social worker recruitment was a challenge faced by councils across Greater Manchester and that it had reduced its rate to 31% since the government statistics were gathered in September last year.

Nadhim Zahawi, the children and families minister, said: “There are more full-time equivalent social workers than ever before, which means there are more dedicated people on the frontline to offer much needed support to some of most vulnerable children and families in the country.

Rise in children taken into care pushes 88% of councils over budget Read more

“Agency staff can play a vital role in managing short-term recruitment needs, fluctuations in demand or to support improvement. We are supporting the recruitment and training of social workers so they have the skills they need for this important job.”

‘It’s become a buyer’s market’

Speaking to the Guardian anonymously, agency social workers said the primary reason they took work as locums was the money. “The money just wasn’t enough for me to keep a roof over my head and pay my bills,” said a social worker who worked for a council in east London before finding work elsewhere through an agency. “I’m not a greedy person. I just want to survive.”

She said her new job as an agency social worker in another London borough paid her around £30 per hour, compared to the equivalent of £17 in her last post. While she reported feeling well supported in her new position, she said the only downside was that she could be let go at short notice. Most agency workers do not receive sick pay, holiday pay or direct pension contributions.

Other than pay and working conditions, she said social workers went to work through agencies in order to save money to do things like travel abroad, pay off debts or build up their professional experience by working for different local authorities.

“It has become a buyer’s market,” she said. “This isn’t a cost-effective way of supporting children and families. The majority [of social workers] are doing it because they really care about the job and are passionate, but they need to pay us a good salary and give us a little bit more care.”