Four men were last night charged with slavery offences, following an investigation into an alleged slave camp at a Travellers' site in Bedfordshire.

The men, all from the same family of Travellers, were charged with conspiracy to hold a person in servitude and require them to perform forced labour.

The investigation into the suspected slave camp has been challenged after nine of the 24 alleged slaves refused to help police with their inquiries. One of the nine accused the police of heavy-handed tactics and described the five arrests made as "a load of rubbish".

The charges came amid reports that the Bedfordshire swoop was connected to earlier raids in four other counties.

In March, 20 people were reportedly rescued from a site in Staverton, Gloucestershire, eight from a site in Enderby, Leicestershire, and seven from a site in Pleasley, Derbyshire. In June, a large group of men were rescued from a site in Hampshire.

Police insist that the four-month undercover operation in Bedfordshire has broken up an "organised crime group", and were questioning 15 alleged victims, who were being treated for malnutrition and other medical problems. Police are looking for two further suspects. One heavily pregnant woman, who is expected to give birth imminently, has been released on bail.

DCI Sean O'Neil, from the Bedfordshire and Hertfordshire major crimes unit, said: "Those people who we continue to help are appreciative of the support that is on offer, but it will take some time to work through with them what has happened."

He said he was confident the operation, called Operation Netwing, had broken up a criminal network.

Police have charged the suspects on slavery and servitude offences under section 71 of the Coroners and Justice Act.

"The new legislation has allowed the investigation more scope and takes into account emotional rather than physical harm," O'Neil said. "I am confident that while the investigation is in its early stages this is a family-run 'business' and is an organised crime group that has been broken up by the Netwing operation."

At the well-established Greenacre caravan park in Leighton Buzzard, which has 16 mobile homes, one of the men taken by police, who did not want to be named, said he had been living in a caravan on a Traveller plot for several years and working as a paver, for which he was being paid £50 a day. "I think it's all a load of rubbish and they just hate Travellers," said the man, who is in his 50s. "Plenty of men who were here wanted to be here and they were getting paid. The police coming in heavy-handed like this is just wrong."

More than 200 police officers raided the site in the early hours of Sunday, aided by a helicopter and dog patrols. Armed officers were also present. Four men and a woman were arrested on suspicion of slavery offences, while 24 men were taken to a medical centre.

Police said the men had been kept as virtual slaves in appalling conditions, forced to work long hours doing physically demanding jobs without pay.

However, the man told the Guardian he had worked for 15 years with one Traveller family who had provided him with work and accommodation when he had nowhere else to go. After refusing to answer police questions, he made his way back to the site. He said: "The police told me I couldn't come back, but I told them it was my home and if I wanted to go back I would go back."

Speaking at the door of her mobile home, one woman – who said she was the wife of one of the arrested men but did not want to be named – said the police claims were "ridiculous".

She added: "The men who were taken were getting paid £30 a day, they had somewhere to live, this is all a load of nonsense."

Police claimed the suspects lured vulnerable men from dole queues and homeless shelters to work at the site. But the woman said they came voluntarily because they knew Travellers would give work to men down on their luck.

"Isn't it better that they have a roof over their head?" she said. "What are they going to do now – when the police have finished with them they will be homeless. It's up to them how they kept their homes, but they could come and go whenever they pleased."

She accused the police of harbouring prejudices against Travellers. "It's complete lies and they are trying to make Travellers look bad. There are two sides to this story," she said.

A police spokeswoman said the 24 men taken from the site were being offered help: "We are giving help to all of the men, but if they do not want it then obviously we are not forcing them to take it."

Paul Donohoe, spokesman for Anti-Slavery International said that, although he could not comment on the details of this particular case, it was not unusual for victims of slavery to resist help from the authorities.

"We do often see the Stockholm syndrome coming into effect – it is not unusual for people who have been 'rescued' to psychologically identify with their enslavers."

Police said on Monday that of the 24 men taken from the site, nine had left the medical reception centre and had chosen not to support the police investigation.

The remaining 15 continue to be assessed for welfare and health needs, and would be interviewed by detectives. Police said it would take a number of days to establish exactly what had happened to them.

Of the men helping police, eight are British, three Polish, one Latvian and one Lithuanian, with two more of unconfirmed nationality. The youngest person to be found on site was 17.