(CNN) Around the same time House Minority Leader Nancy Pelosi was declaring she had broad support to remain the top Democratic leader and jabbing back at her critics, a group of her colleagues met privately to brainstorm on whether there was a way to force her out.

New York Democratic Rep. Kathleen Rice, one of a small group who has gone public with the message that Pelosi should go, hosted a dozen Democrats in her office Thursday for an hour-long strategy session.

Rep. Cedric Richmond, the Chairman of the Congressional Black Caucus, who many have touted as a rising star in the party, attended the session, according to three Democrats who also attended. The CBC includes roughly 40 House Democrats, many in senior leadership and committee positions, including Pelosi allies.

In an interview on CNN before the meeting, Rice admitted that the problem facing the party now is that no one has emerged as an alternative.

When pressed to give a name of who might fill that void, Rice said, "I look to a lot of my colleagues now. We have a lot of talent in the Democratic Party," but she pointed out that "we don't have an infrastructure in our caucus allowing more voices to be heard, and that's the problem."

JUST WATCHED Pelosi: GOP health bill is mean and heartless Replay More Videos ... MUST WATCH Pelosi: GOP health bill is mean and heartless 01:37

"Obviously people are very concerned about where we are and they want to have a conversation about where we need to be," Rep. Tim Ryan of Ohio told reporters about why the group got together. Ryan made it clear after his failed bid to oust Pelosi in the fall that he wasn't planning to challenge her again and was just supporting the effort.

Massachusetts Democratic Rep. Seth Moulton, another outspoken critic, also attended, according to sources -- and some in the group continue to point to the Iraq war veteran as a young member who could be cultivated to run.

Another Democrat who has been pushing for leadership changes, Rep. Filemon Vela of Texas, told CNN the group didn't make any decisions at Thursday's meeting, but those who participated believe there is growing unrest among House Democrats about the direction of the current leadership team.

"I think there was consensus within the room that there are other members within the caucus who feel just like we do," Vela told CNN. He emphasized that the group "was a diverse group from an ideological, geographic and ethnic standpoint."

Pelosi insisted that the current leadership structure already gives opportunities to younger members, telling reporters, "I have always featured the young 30-somethings ... and they are so impressive. So, we are paving a way for a new generation of leadership."

But she made it clear that while she heard the pushback from the small group, she was looking to the broader caucus to back her up.

"I respect any opinion that my members have but my decision about how long I stay is not up to them."

Pelosi didn't give names when she referred to those who want her to step aside, but she called out those chose to go on television to criticize her, saying "have your fun."

Ryan, who has appeared on multiple networks to complain about the current direction, shot back, "this isn't fun for any of us."

Vela said he believes if the party sticks with the Pelosi in the top slot, it will hurt the efforts in the kinds of districts they need to flip to win back control of the House in 2018.

"As long as Leader Pelosi is perceived as the leader of the House Democratic Caucus Republicans are going to continue to spend millions and millions of dollars in those swing districts to convince those swing voters, those independent voters, those Republican voters who might go our way, not to vote for our Democratic candidate because of Leader Pelosi," Vela said.

The group who met Thursday decided to get together again soon, according to attendees.