(CNN) The college admissions scandal reinforced for many what they have long believed: That the process can be gamed by those with wealth and influence.

It has spurred discussions about why factors such as donations, athletics and legacy status are baked into the admissions process, which has traditionally benefitted wealthy families. Yet affirmative action, which is intended to help underrepresented minorities, gets intense scrutiny and legal challenges.

"Some people have said wealth is affirmative action for white people," said Anthony Jack, an assistant professor of education at Harvard University.

It is not affirmative action that threatens the fairness in the college admissions process, its supporters say, but rather the advantages of the rich and powerful.

"These families exposed a system and I hope (this) injects into the American imagination just how much money and backdoor ways that wealthy, especially white wealthy families, get into college and universities," Jack said.

JUST WATCHED Fmr. Harvard Pres: Not wrong to say elites rig system Replay More Videos ... MUST WATCH Fmr. Harvard Pres: Not wrong to say elites rig system 04:35

Larry Summers, the former Harvard University president, told CNN's Christiane Amanpour that there needs to be a lot of soul-searching in higher education.

Americans are "not entirely wrong" to think "that elites are rigging the system for their own benefit and for the benefit of their families," he said.

Discourse around affirmative action

The scandal has hit a nerve, going beyond discussions about elitism and raising issues of race.

"Imagine believing it's affirmative action that's the problem with college admissions," the ACLU tweeted.

Imagine believing it's affirmative action that's the problem with college admissions. https://t.co/e3FEmKiLCs — ACLU (@ACLU) March 12, 2019

Hey Guys: Remember when some alleged that affirmative action was the biggest threat to the integrity of the college admissions process. — Erica L. Green (@EricaLG) March 12, 2019

"Affirmative Action: White Privilege Edition," starring Aunt Becky and Lynette 👇🏽 https://t.co/DakMXaEqrA — Jose Antonio Vargas (@joseiswriting) March 12, 2019

Jack says there is a connection between the scandal and affirmative action.

"It's so written into the American imagination that these spots (at prestigious institutions) are for white people and anytime a black student or a Latinx student gets in, it's taking a spot away from them. That's not what's happening," said Jack, the author of "The Privileged Poor," about the experience of low-income students at elite universities.

"What this is exposing," he said of the scandal, "is the steps and the leaps and bounds that wealthy families take to secure a spot that is rightfully no one's -- that they think they have proprietary ownership of."

Many underrepresented minorities say they're constantly having to prove themselves and their qualifications for a spot.

A person of color has to "prove yourself at every turn," said Tiffany Cross, the co-founder and managing editor of The Beat DC, in a panel discussion on "CNN Tonight" with Don Lemon.

"You can be an Ivy League graduate and show up to the table and somebody's going to question your existence there. Nobody ever asked, 'How did this basic, ordinary person next to me, who isn't a person of color, earn their space here?'"

JUST WATCHED Does it matter which college you go to? Replay More Videos ... MUST WATCH Does it matter which college you go to? 01:56

Yet there is little discussion about underqualified white students who benefit from preferences in the admissions process such as sports, family influence and legacies, some say.

Legacies are applicants who are regarded preferentially because they are the children of alumni. They also tend to be white and wealthy, wrote Daniel Golden, the author of the 2006 book "The Price of Admission: How America's Ruling Class Buys Its Way into Elite Colleges."

Athletes of patrician sports, such as sailing or water polo, are recruited to college athletics. These types of sports aren't accessible for students from inner-city schools.

"It shows the hypocrisy of the elevated status that legacies and athletes get in the admissions process," Jack said. "There's no moral, social or political justification for those two groups to get preferential treatment in admissions."

These preferences benefit mostly white and wealthy applicants, but aren't part of the ongoing debate about affirmative action and what should be considered in the admissions process, he said.

Meanwhile affirmative action remains the target of several lawsuits, including a pending case against Harvard, that was brought on behalf of several Asian-American students by a group called Students for Fair Admissions. The group was set up by Edward Blum, a longtime opponent of affirmative action who in the past has used white plaintiffs to challenge racial policies.

JUST WATCHED Don Lemon: Trump was right. The system is rigged Replay More Videos ... MUST WATCH Don Lemon: Trump was right. The system is rigged 03:43

The group argues that Harvard disfavors high-achieving Asian-Americans and gives a boost to African-American, Hispanic and other traditional beneficiaries of affirmative action.

"They're really a conservative group who is using the face of Asian students to say that they're taking admissions access from Asian students to give them to black and brown people," Cross said during a Tuesday panel discussion.

"It's interesting that the privileged people will have other people of color thinking that we stole something from them. They're not going after privileged people."