Most of us are well aware that sexual harassment was a problem in America well before the Harvey Weinstein scandal erupted this summer. But there is no denying the Weinstein story has taken the wraps off of a massive number of alleged and confirmed cases of this kind of misconduct in Hollywood.

Now, the same thing could happen on Capitol Hill as the John Conyers case threatens to blow a longstanding congressional tradition of unethical secrecy and subterfuge sky high.

Just to recap, it first came to light earlier this month that Rep. Conyers signed off on a $27,000 payout to a former staffer who says she was fired for resisting the congressman's sexual advances. Conyers acknowledged the 2015 payout after details of it leaked online. But he has denied he sexually harassed the employee. Conyers has been resisting calls to resign, but Sunday he did step down as the ranking member of House Judiciary Committee.

But wait, there's more. So much more. First off, Conyers has publicly made himself out to be a victim. In a tweet over the Thanksgiving weekend, he focused his anger on the fact that this story became public and blamed it on a "alt-right blogger":

Instead of apologizing for using his congressional office funds to secretly settle a harassment case, Conyers is angry the public has found out. Even if the public is the source of the cash used to pay off his accuser.

But here's the real reason this could blow up: Because of this story, we now know that the same Congressional office where this complaint was filed against Conyers has settled a total of 264 complaints to the tune of $17.2 million of taxpayer money over 20 years. To be specific, that would be the U.S. Congress Office of Compliance. And so far, we don't know the identity of any of the others named in the complaints.

That cover could soon be blown for two reasons. First we could see a scenario unfold where, like the sexual harassment incidents in Hollywood, people who settled cases with the Office of Compliance will now feel emboldened to speak out.

Second, Conyers could start taking down his colleagues if he begins to feel even more threatened. That's what it sounded like when his attorney Arnold Reed issued a statement explaining why Conyers would not resign. Included in Reed's statement was a sentence that sounded like a threat:

"If people were required to resign over allegations, a lot of people would be out of work in this country including, many members of the House, Senate and even the president."

On the surface, that quote could simply be a rhetorical point about how so many other politicians in the past and present have faced similar allegations and managed to get elected and/or stay in office. That list would include President Donald Trump, former President Bill Clinton, and, (for now), Senator Al Franken.

But the other not-so-hidden message is that Conyers and his attorneys have information on many other members of Congress from both parties who have had similar allegations made against them and settled with taxpayer money through that secretive Office of Compliance.

That would be a threat to whomever was the subject of those 264 total settlements, but more like a tantalizing promise for the rest of the country.

To be clear, as a citizen, Conyers was certainly deserving of due process to defend himself against allegations of any kind and affix a burden of proof on his accusers. And contrary to House Minority Leader Nancy Pelosi's protestations on "Meet the Press" Sunday, this case proves he has most assuredly been given that.

The problem is that Conyers and his colleagues are not ethically entitled to the use of taxpayer funds and congressional facilities to keep matters quiet and out of court.

If this story rightfully results in the continuing demand to unseal the details of those other 264 complaints, then you have something that could ensnare a lot of sitting members of Congress. Even if those 264 complaints hit just 20 to 30 current senators and representatives, that could make a huge impact on Congress for a number of reasons.

First off, it could speed up a change in the partisan balance of power. Remember, the Senate Republican majority is just held up by a paper thin 52-48 margin. The Republican majority in the House is 45 seats. But that's not an insurmountable number going into the 2018 midterm elections based on the latest polls. Of course, only the folks at the Office of Compliance know which party and which siting members have been targeted with the most of those complaints. This is the kind of scandal that could set off mass resignations that give the Democrats majority power. Or it could also help the GOP gain a bigger majority in order to more easily pass struggling bills like tax reform and the on-again, off-again Obamacare repeal.

This kind of scandal has helped defuse congressional powerhouses before. When Oregon Republican Senator Bob Packwood was forced to resign over sexual assault allegations in 1995, his seat in that blue state went to Democrat Ron Wyden who still holds it today. What many people forget was that Packwood had been a strong debater in the Senate and was credited with being a key player in dismantling the Clinton administration's efforts at a universal health insurance bill. In other words, sexual harassment allegations can bring down some big and powerful people. In an interesting twist, the most powerful pressure on Packwood that ultimately forced his resignation came from his fellow Republican senator and now Senate Majority Leader Mitch McConnell.

Secondly, this could lead both parties to get more serious about supporting women candidates in congressional elections. It's a simple fact that almost no women are ever accused in these kinds of sexual harassment and backing more of them in this environment seems like a better political bet. That could lead to the biggest changes of all, considering the fact that women still make up only about 20 percent of the House and Senate combined.

It sure seems like that kind of upheaval is frightening even some supposedly outraged folks in Congress. Yes, Rep. Jackie Speier and Senator Kirsten Gillibrand have co-authored a bill to get the Office of Compliance to change its practices and name every member of Congress accused in filed sexual harassment claims. That would be a good thing. But even their bill is not calling for past complaint and settlement records to be unsealed. And that speaks volumes about how closely these secrets are held and how valuable they are to the public and those journalists endeavoring to uncover them.

Only the leadership of both parties in Congress is likely to know the breakdown of just which side would suffer the most if all that information became public. Perhaps that's why Pelosi defended Conyers so vigorously. That's even though a Republican in Congressman Joe Barton, is facing heavy scrutiny now after nude pictures of him surfaced online. And of course, Republican senatorial candidate Roy Moore is under extreme pressure from members of both parties to drop out of the race to succeed Jeff Sessions in Alabama. I've heard both Democrat and Republican pundits confidently guess their side will come out on top in this scandal-laden numbers game.

But you don't need to be a political genius or odds maker to see how this could upset the balance of power and/or supercharge a women's movement in Congress. And it all really hinges on the unsealing or leaking of those Office of Compliance records.

Until that happens, all we know is that the Conyers scandal could turn Capitol Hill completely upside down. But if the way his case was handled is any indication, it's hard to argue that Capitol Hill doesn't completely deserve it.

Commentary by Jake Novak, CNBC.com senior columnist. Follow him on Twitter @jakejakeny.

For more insight from CNBC contributors, follow @CNBCopinion on Twitter.