Miami, Florida (CNN) -- Their stories were chilling: Students at a reform school recounted beatings and sexual assaults at the hands of school administrators and other employees who were supposed to be taking care of them.

But a state investigation into the claims -- which date back to the 1950s and 1960s -- found insufficient evidence to prosecute any former workers at the Dozier School for Boys in Marianna, Florida.

Results of the investigation, which was conducted by the Florida Department of Law Enforcement at the request of Gov. Charlie Crist, were released Thursday. The FDLE investigated allegations made by a group of men who said they were beaten and sexually abused when they attended the school about five decades ago.

Read the FDLE's final report

State Attorney Glenn Hess told the FDLE he would not pursue any prosecutions in a letter dated February 25.

"The primary obstacles to bringing criminal cases based on events alleged to have occurred more than 40 years ago are the Constitutional rights to a speedy trial and to due process," Hess wrote.

"Florida's statute placing time limitations on prosecutions must also be considered. In a nutshell, citizens are protected from being prosecuted for crimes that occurred so long ago that preparing a defense would be difficult or impossible. The claims provided here are an example," he said.

The allegations against school workers were made by a group of men -- now in their 60s -- who call themselves "the white house boys" after a nondescript white concrete building on the school grounds where they say the beatings and torture were carried out, often with a thick leather strap.

One member of the group, Dick Colon, told CNN the outcome of the investigation shows "Florida is running with a scam and everybody's behind it."

"FDLE is whitewashing this thing like you wouldn't believe," said Colon, who was interviewed by the FDLE in the investigation.

Colon said he was sent to the school for stealing cars, and went to the "white house" 11 times during his 30 months there, where he suffered unimaginable beatings. He remembers them as if they happened yesterday, he said.

Colon said he was forced to lie face-down on a blood-soaked pillow that had small pieces of lip, tongue and skin from the previous boy's beating. He'd clench the metal bar of the bed as the strap struck him.

"After that tick, you'd go 'Aaaahh,' and then you'd grab that bar, and go 'Ooooohhhhhhh,' and the spindles of the bed would bounce, and sometimes the bed would come off the ground," Colon told CNN.

The FDLE said its investigators interviewed six former staff members from Dozier and more than 100 former students and their relatives. But the agency acknowledged that most staff members who could have provided information have died.

The report said there were inconsistent accounts from the men about various incidents, and a forensics examination of the "white house" building found no evidence of any blood.

Most of the former students interviewed during the investigation agreed that a wooden paddle or leather strap was used for punishment, the report said, but there was disagreement about the number of spankings and the severity.

"There was little to no evidence of visible residual scarring," the report said.

"With the passage of over 50 years, no tangible physical evidence was found to either support or refute the allegations of physical or sexual abuse," it added.

Hess cited another example in his letter, in which a former student said he had been sexually assaulted in a "rape room," but then admitted he was not sure he had been raped.

"Time has blunted even the accuser's memory. Due process demands that the accused be informed of the charge he is to answer with specificity. These claims do not suffice," Hess wrote.

Crist also asked the agency to look into 32 unidentified graves on the school grounds. Some of the accusers said the graves -- marked only by white steel crosses, rusted with the passage of time -- contained the bodies of boys who were beaten to death by school workers.

But the FDLE investigation announced findings in 2009 that it had accounted for all the students.

"Enough information has been corroborated on who is buried there," Mark Perez, chief of Investigations for the FDLE, said in 2009. "We went and identified all of the individuals who perished while in custody."

Colon said he reacted to the beatings and other events he witnessed by burying the pain inside. He also said another boy's terror has left him wrestling with his own best and worst instincts.

He said he walked into the school's laundry room one day and saw a black teenager inside a large tumble dryer that was running. He wanted to save the boy, Colon said, and tried to talk himself into being brave.

"I said, 'Do it! Do it! Do it!' " he recalled, his eyes brimming with tears. "And then I thought to myself, 'If you do it, they're gonna put you in there. You're gonna be next.' And I walked away."

But the FDLE report said there were inconsistencies in the story and a lack of evidence.

"They have no evidence, they say, of any deaths?" Colon said angrily. "When I told them I came out and saw this black boy in the dryer -- did I witness a death or a goddamn birthday party?"

About 300 of the former reform school students sued former school administrators. The lawsuit was recently dismissed.

"We're attempting to move forward with a claims bill in the Florida state legislature to try and achieve justice for our clients," said Gregory Hoag, one of the students' attorneys.

One of the men named in the original lawsuit is Troy Tidwell, a former reform school administrator, who still lives near the school grounds in Marianna.

Tidwell refused to cooperate with the FDLE investigation, the agency's report said, because his cooperation could result in a criminal prosecution.

Now in his 80s, Tidwell did give a videotaped deposition in the civil case last year. He said the boys at the school who misbehaved were "spanked."

"Never was a boy beat in my presence," he said.

Tidwell's attorney did not respond to CNN's calls for comment.

Colon says the FDLE's report about what happened at the school is wrong, and he wants to do his own investigation.

"We're going to get some ground-penetrating radar, and we have guys who will go and find the bodies," he said.