Lori Loughlin and husband Mossimo Giannulli don’t appreciate the backlash over their alleged involvement in the college admissions cheating scam.

After pleading not guilty earlier this week, a source tells PEOPLE that the couple resents how the case is playing out in the public eye.

“This is putting unspeakable stress on her and her family,” a source close to Loughlin tells PEOPLE. “They’re having to play this all out publicly, and they’re fair game for jokes and memes, but also outraged [by] people who are saying that they are cheaters.”

“They’re being destroyed,” the source adds.

On Monday, the couple pled not guilty to both charges they face in the case: mail fraud and money laundering conspiracy. A source previously told PEOPLE that the plea was the “only choice” they had after rejecting an offer for a deal from prosecutors.

But with the plea now comes the possibility of a trial.

“The idea of going to trial is terrifying for Lori,” says the source close to the actress. “Everything comes out in trial, whether or not it’s relevant to the case. She will be under a microscope, and you only have to look at the paparazzi outside the court to know that there is widespread interest in this case. She will lose every bit of her privacy, and that’s a shame. This really is a family matter.”

If convicted, Loughlin and Giannulli face up to 20 years in prison for each charge.

RELATED: Lori Loughlin and Husband Plead Not Guilty in College Admissions Cheating Scandal

On March 12, the U.S. Attorney’s Office in Massachusetts announced that it had charged 50 people, including Loughlin and fellow actress Felicity Huffman, in the cheating scandal. The two actresses, along with coaches, admissions counselors and parents were accused of such alleged crimes as falsifying SAT scores and lying about the athletic skills of their children.

Huffman has agreed to plead guilty to one count of conspiracy to commit mail fraud and honest services mail fraud, saying, “I am in full acceptance of my guilt, and with deep regret and shame over what I have done, I accept full responsibility for my actions and will accept the consequences that stem from those actions.”

Splash News Online More

RELATED: Amid the College Admissions Scandal, Here Are 8 Organizations Helping Disadvantaged Students