Three weeks after Jeffrey Epstein was arrested on child sex trafficking charges, Democratic fundraising groups have remained silent on whether they will donate or give back contributions from the wealthy financier.

Epstein started giving to these organizations just over two decades ago. He was first accused of sex crimes in the mid-2000s. When asked whether the Democratic National Committee or the Democratic Senatorial Campaign Committee will return or contribute these Epstein funds to a charity organization, press representatives for the groups did not respond. CNBC has reached out to the committees numerous times over the past two weeks.

Epstein gave at least $80,000 combined to the DNC and the DSCC from the late 1990s through the early 2000s, according to Federal Election Commission records. Epstein's donations to the committees were given either directly or through Democratic joint fundraising operations, including one in part led by Hillary Clinton's U.S. Senate exploratory committee called New York Senate 2000.

Epstein has pleaded not guilty to the charges.

Don Fowler, who was DNC co-chairman from January 1995 through January 1997, questioned why the committees should give away these donations due to the checks being written decades earlier.

"Go back and give money that he gave 20 years ago? Are you nuts? That's my answer to that," Fowler told CNBC in an interview Wednesday.

"Maybe the RNC [Republican National Committee] should return any money that anyone gave to supporters of Richard Nixon," he emailed later.

When asked about Democratic Senate Minority Leader Chuck Schumer, who gave $7,000 in campaign contributions from Epstein to charities, Fowler said the New York lawmaker is "one person who is completely isolated from any political threat." He called it "a nice little symbolic gesture."

The Democratic Congressional Campaign Committee saw a $10,000 donation from Epstein last year but immediately sent it back.

Stacey Plaskett, a Democratic congresswoman, recently reversed course and gave the value of her checks to charities.

While Epstein has given to Republicans in the past – including former President George H.W. Bush, retired Sen. Bob Dole and former Ohio Gov. John Kasich, who received a $1,000 Epstein contribution during his time in Congress – records do not show that he gave to GOP fundraising committees.

Epstein was listed on a guest roster at a DNC donor event in 1995 at the Palm Beach, Florida, home of billionaire political donor Ronald Perelman, which included the likes of President Bill Clinton, singer Jimmy Buffett and actor Don Johnson.

The Palm Beach Post, which reported on the gathering at the time, said that some attendees gave as much as $100,000 to the DNC to dine with Clinton. Another participant was Arnold Paul Prosperi, a longtime college friend of the then commander in chief. Prosperi conducted regular visits to a Palm Beach County, Florida, jail where Epstein resided after his initial 13-month prison sentence in 2008.

Fowler reportedly participated in the Perelman dinner, and when asked about this, he says he can't recall meeting Epstein and didn't remember anything about the gathering.

Democratic and Republican committees in the past have come under scrutiny for how they've handled the contributions they've received from those later accused of some forms of sexual harassment.

After Hollywood movie producer and Democratic megadonor Harvey Weinstein was accused by multiple women of harassment and assault, the DNC gave the $30,000 it received from him during the 2016 election cycle to three political groups instead of charities. Republicans, at the time, pounced on the lack of money going to charitable organizations and that the refund was only a fraction of the at least $300,000 the DNC had brought in from Weinstein since he started getting involved with political giving. Weinstein had been giving to Democratic candidates and their committees since 1991, records show.

Weinstein has denied any incidents of nonconsensual sex.

The RNC, likewise, has received its share of criticism after deciding to continue to accept hundreds of thousands of dollars from casino magnate Steve Wynn, the group's former finance chair, after he has seen numerous accusations of sexual misconduct.

In the second quarter, FEC records show that the RNC saw almost $245,000 added to its coffers from Wynn. The National Republican Senatorial Committee and its affiliated joint fundraising group, NRSC Victory, combined saw $290,000.

Republican officials have said they have no intention of giving away or return those funds.

Wynn has denied allegations of misconduct.