Washington (CNN) -- The Senate Ethics Committee announced Thursday that it has referred key findings of its case against former Sen. John Ensign to the Justice Department, laying the groundwork for possible prosecution of the Nevada Republican.

Ensign resigned his Senate seat May 3 after revelations that he had an affair with a female aide who was married to another top aide and that his parents subsequently gave money to the aides' family.

The special counsel investigating the case "respectfully submits that there is substantial credible evidence which provides ... cause for the Committee to conclude Senator Ensign violated Senate Rules and federal civil and criminal laws, and engaged in improper conduct," a report concluded.

"The committee has reason to believe laws were broken," said California Sen. Barbara Boxer, the Democratic co-chairwoman of the committee. "The Justice Department will look at it. They have to."

The report cites "credible evidence" that Ensign and his parents made "false or misleading statements" to the Federal Election Commission and engaged in a potential obstruction of justice, among other things.

Ensign has admitted having an affair with Cindy Hampton, a campaign aide and the wife of Ensign's former chief of staff, Doug Hampton. Investigators have been examining efforts by Ensign to assist the Hampton family by providing a nearly $100,000 payment to them, arranging lobbying work for Doug Hampton and possibly meeting with Doug Hampton on a lobbying matter in violation of Senate rules.

Doug Hampton was indicted this year on charges related to an alleged violation of federal conflict of interest laws.

The ethics committee announced the appointment of a special counsel for the Ensign investigation in February. According to Ensign, the Justice Department has told him it would not bring criminal charges in the case.

"I am gratified that, after extended investigations, both the Department of Justice and the Federal Election Commission saw no grounds on which to charge me with improper conduct," Ensign said in statement released when he resigned. "I was hopeful that, with the closure of these investigations against me the wear and tear on my family and me would soon be over. This was not the case."

Ensign's attorney released a statement Thursday stating that the senator was "disappointed and confused" that the committee issued its report without considering a point-by-point rebuttal submitted by his legal team on Wednesday.

The rebuttal concluded with an insistence that Ensign "is not proud of his conduct nor is he asking that his improper conduct be ignored."

But "acknowledging one's errors ... is not the same as admitting he had the intent to violate the law -- this he did not do."

Ensign, 53, was elected to the Senate in 2000. Nevada GOP Gov. Brian Sandoval recently appointed fellow Republican Dean Heller to serve the remainder of Ensign's term, which expires in January 2013.

Heller had been representing Nevada in the House of Representatives.

CNN's Alan Silverleib, Dana Bash, Kate Bolduan, Bill Mears and Tom Cohen contributed to this report.