PROVIDENCE, R.I. -- U.S. colleges trying to respond decisively to complaints of sexual assault are getting slammed with lawsuits from men who say they've been unfairly suspended or otherwise punished.

At least 75 men have sued their schools since 2013, complaining largely of reverse discrimination and unfair disciplinary proceedings.

Most of the men were never charged with a crime because the accuser didn't go to police or authorities decided there wasn't enough evidence.

Get Breaking News Delivered to Your Inbox

Schools say they feel caught in the middle.

On one side is the U.S. Education Department, which is demanding colleges deal firmly with reports of sexual assault. On the other side are students who say they are not getting a fair shake.

The day after the NCAA revealed a slot for Yale in March Madness for the first time in more than 50 years, the team's expelled captain, Jack Montague, announced his plans through his lawyer to sue the school for making him "a whipping boy."

The Yale Daily News reported Montague's lawyer, Max Stern, released a statement to local media last week detailing plans to sue.

"Last week, the media widely reported on statements made by Yale students and posters put up on campus which condemned Jack Montague directly as the named culprit and as a rapist, thus slandering him with this accusation. He was never accused of rape and Yale took no steps to correct these actions. As a result, Mr. Montague has no choice but to correct the record," the statement read.

According to multiple reports, the incident in question involved Montague's relationship with an unnamed female student. According to Stern's statement, a university fact-finding committee found that he had had sexual relations with the woman on four occasions. He was allegedly expelled for what she claimed was unwanted contact during the fourth encounter. Montague has said it was consensual, and his lawyer points to her returning to the room afterward as proof.

The Yale Daily News writes: "A year after the fourth encounter, which occurred in October of 2014, the female student reported the incident to Yale's Title IX coordinator. A Title IX official, not the student, filed a formal complaint within the University-Wide Committee on Sexual Misconduct."

Montague believes Yale's actions were "wrong, unfairly determined, arbitrary, and excessive by any rational measure," according to his lawyer, who added: "We believe that it defies logic and common sense that a woman would seek to re-connect and get back into bed with a man who she says forced her to have unwanted sex just hours earlier."

In a statement, a Yale spokesperson defended the school's investigation.

"The allegations are investigated by an impartial fact finder, heard by five trained members of the Yale community, and decided by the accused student's dean. Throughout the process, all parties have advisors, which can be legal counsel, and they can appeal a decision," the statement read. "Where cases involve judgments about the witnesses' credibility, all of the available corroborating or contradictory information is carefully weighed to determine who is telling the truth."

After the team wore T-shirts to the Ivy League championship game, with Montague's nickname on the back, posters appeared on campus warning the Bulldogs to "stop supporting a rapist."

"To show support in the most public and provocative way possible, you know it was a nationally televised game. I think that's very, very irresponsible," said Helen Price, the director of Unite Against Sexual Assault Yale.

The team later apologized.

In September, a report released revealed a quarter of women surveyed at more than two dozen universities say they experienced unwanted sexual contact sometime during college.

The survey was sent to nearly 780,000 students at the association's member colleges, plus one additional university. About 150,000 participated in the online questionnaire.

The results were generally in line with past surveys on sexual assault and misconduct on college campuses -- and confirmed that alcohol and drugs are important risk factors.

Researchers cautioned against generalizations, partly because of different rates among women and men and students at different years in their schooling. It was not a representative sample of all the nation's colleges and universities.

Some students attended schools that have recently grappled with reports of sexual assaults or misconduct, including the University of Virginia, the University of Wisconsin-Madison and Ohio State.

Report came a few months after court documents showed that three University of Virginia graduates and members of a fraternity profiled in a debunked account of a gang rape in a retracted Rolling Stone magazine story were suing the publication and the article's author.

The three graduates filed suit in July in U.S. District Court in New York. They are also suing Rolling Stone's publisher, Wenner Media.

Rolling Stone asked for the independent review after other news media organizations exposed flaws in the November 2014 article, titled "A Rape on Campus." By Dec. 5, Rolling Stone apologized and acknowledged discrepancies in the article.

A four-month police investigation produced no evidence that the attack occurred.

A federal judge in Rhode Island last month allowed a case to move forward by a Brown University student suspended for 2 1/2 years over a sexual assault accusation.

The get-tough approach by colleges is attributed largely to a 2011 letter from the U.S. Department of Education's Office for Civil Rights. The letter told schools they must promptly investigate allegations of sexual assault and harassment, even if the accuser does not make a complaint to the institution.

Schools that do not comply can face an investigation and a cutoff of federal money. As of mid-March, the Office for Civil Rights was conducting 219 such investigations at 173 schools.

Known as the "Dear Colleague Letter," it has been hailed by advocates who say many schools are now moving in the right direction to address campus sexual assaults.

"For a very long time, there was no due process for victims. Victims were told to withdraw from school. Victims were told to take the semester off," said Colby Bruno of the nonprofit Victim Rights Law Center. Now, "yes, there are more decisions against perpetrators. Yes, perpetrators are being held accountable. And that is going to bother people."

Advocates for the accused say that school disciplinary panels are unequipped to handle such serious allegations and that colleges have gone so far to accommodate alleged victims that they are trampling on the rights of the accused.

Under the federal guidance, when a school learns of allegations of sexual assault or harassment, it must take immediate steps to ensure the victim's learning environment is free of hostility.

Some critics have said that universities should not be handling such cases at all and that sex crimes should instead be reported to police.

But victims' advocates and many schools oppose that, saying it would discourage women from coming forward. Also, police investigations can take months, and prosecutions longer, meaning some cases might not be resolved before graduation.

Andrew Miltenberg, who represents the Brown student and close to 100 other male students accused of campus sexual assaults, said schools have become "hyper-aggressive" since the Dear Colleague Letter.

A common thread in his cases, Miltenberg said, is that they had some element of a consensual encounter, in which the two were dating or knew each other. Alcohol is often involved, sometimes a year or more has elapsed, and there are rarely witnesses.

Miltenberg represents Paul Nungesser, who sued Columbia University, saying it violated his rights when it allowed fellow student Emma Sulkowicz to obtain class credit for her "Mattress Project," in which she carried around the mattress on which she said he raped her. Nungesser denied the accusation and was found not responsible by the school disciplinary process.

Emma Sulkowicz, a student at Columbia University, where she has been carrying a mattress with her in a protest after she says she was sexually assaulted on the campus in New York City The Columbia Spectator via CBS New York

Nungesser said in his lawsuit that he was branded a "serial rapist" and forced to return to his native Germany because he couldn't get a job in the U.S. A judge this month dismissed the lawsuit but said Nungesser may refile some claims.

Sulkowicz went to police with her allegation against Nungesser, but they did not bring charges. She has said the campus disciplinary process was badly mishandled.

In a 2014 essay in Time magazine titled "My Rapist Is Still on Campus," she wrote of crying and hyperventilating when she learned he had asked permission to work in the darkroom during a photography class she was taking.

"As long as he's on campus with me, he can continue to harass me," she wrote.