As incoming House Speaker Nancy Pelosi works to consolidate votes for her leadership bid, a group of largely centrist House Democrats have remained holdouts. Publicly, they’re pushing for tweaks to House rules once Democrats take over the chamber in January. But behind the scenes, a leading advocacy group that has helped organize this coalition of moderates has been itching to make life hell for Pelosi.

Internal communications reviewed by The Daily Beast show that early this year the group No Labels, a centrist advocacy organization, contemplated a plan to kneecap Pelosi’s political standing. In one exchange, a top official with the group even laid out the pros and cons of turning the California Democrat into a “bogeyman.”

That was well before Democrats took back the House of Representatives. In the weeks since they reclaimed congressional power, Pelosi has sought to earn back the Speaker’s gavel. And though the vast majority of her caucus supports her bid, moderate Democrats allied with No Labels remain some of the few party members refusing to give her their votes absent concessions from Pelosi and her team.

No Labels has publicly couched its efforts in conciliatory language, posting items on its website that subtly nudge other Democrats towards a Pelosi challenge, or that gently suggest it might be time for a leadership change. The group also wants Pelosi to commit to weakening the power of party leaders and committee chairs, among other measures that it says will empower rank and file members and reduce gridlock.

Behind the scenes, No Labels and its leader, political strategist Nancy Jacobson, have been more skeptical of Pelosi and more willing to try and marginalize her among her members.

Emails obtained by The Daily Beast show that No Labels leadership contemplated a campaign to attack Pelosi aggressively after the primary campaign of centrist Rep. Dan Lipinski, who faced a primary challenge this year from Marie Newman, a progressive political neophyte. Lipinski’s pro-life stance had alienated a number of Democrats, but he was a proud member of the No Labels-backed House Problem-Solvers Caucus, and the group worked through a network of allied super PACs to support his reelection bid.

“Nancy, I have been thinking about our using Pelosi as the chief bogeyman in our messaging post-Lipinski,” began one email, subject line: “Pelosi as bogeyman.”

Pelosi had endorsed Lipinski. But No Labels leadership was convinced that her support was a fig leaf. Jacobson, according to a source familiar with the group’s internal deliberations, was convinced that Pelosi had secretly tried to scuttle the congressman’s reelection and proposed publicly attacking the Democratic leader in the run-up to the midterms.

“We were trying to figure out, assuming we got a positive result [in the Lipinski race], which we did, what would be the comms strategy afterwards,” said the source. “Nancy Jacobson’s immediate answer was, ‘I want to make Pelosi the bogeyman.’ She wanted to make it all about Nancy Pelosi and how she was going after incumbent Dems. None of that was true.”

According to the emails, No Labels chief strategist Ryan Clancy appears to have tried to talk Jacobson down. A direct confrontation with Pelosi would blow back on the group’s congressional allies, he explained. It would also be unprecedented; No Labels had never engaged in similar campaigns against congressional leaders of either party. Clancy instead proposed to make Sen. Bernie Sanders (I-VT) the object of the group’s criticism. Unlike Pelosi, Sanders had actually endorsed Newman; had no procedural power to wield against members of the Problem-Solvers Caucus; and wasn’t even a Democrat.

“This is us, in effect, declaring war on Pelosi,” Clancy warned. “No Labels has never identified a party leader as ‘the enemy’ before. We didn’t do it with Obama or Trump. We haven’t done it with Schumer, Ryan, or McConnell. If we do this, we should consider some very real downsides.”

At the end of his memo, Clancy conceded that, “at some point,” No Labels “is probably going to go to war with Pelosi. And it probably should.” But “I don’t know that now is the time to do it, especially when we have a perfectly good villain to use in Bernie.”

In an email statement to The Daily Beast, Jacobson said she was not aware of any “internal communications” about a campaign against Pelosi. She also noted that the group had “ never made any decision to run an anti-Pelosi campaign and are NOT running one. We don’t care about personalities we do care about the process. We just want the rules changed regardless who was/is to become leader. As you know, rules get changed at the start of every new congress not in the middle.”

However, Jacobson did note that No Labels had “no relationship with Leader Pelosi but we do have a strong relationship with Senator Chuck Schumer who has met with our group numerous times and who many of our members support.”

Ultimately, No Labels appears to have heeded Clancy’s advice. A day after Lipinski’s victory, the group’s senior adviser Margaret White penned a column on the race for The Hill. The column didn’t mention Pelosi. Instead, it dubbed Lipinski “a Democrat under attack from the far left for failing to embrace every element of Bernie Sanders’ extreme agenda.” It also chided critics of Lipinski by noting that he had earned Pelosi’s support.

But the group’s distaste for Pelosi, however, did not dissipate. They’ve tweeted about her diminished public support.

And since Democrats won back control of the House this month for the first time in eight years, No Labels has egged on lawmakers—including Lipinski—to withhold their support for Pelosi’s speakership bid in an effort to extract some procedural concessions.

Though it all, No Labels insists that Pelosi is not the “boogeyman” its leader had contemplating portraying after Lipinski’s primary victory. “No Labels is not against Nancy Pelosi or any other speaker candidate,” the group claimed over the weekend. “We are FOR rules changes that empower members in both parties who want to work across the aisle to find solutions and prevent the fringes–in both parties–from perpetuating endless gridlock.”