Story highlights Some Democrats would like to have Nancy Pelosi leave her leadership position

Pelosi didn't directly respond to Rice's call, but stressed for Democrats to come together

(CNN) A small group of House Democrats are publicly saying it's time for House Minority Leader Nancy Pelosi to step aside as the face of the party ahead of the 2018 midterms following Tuesday night's disappointing loss in a special election for a Georgia House seat.

"We need leadership change," New York Democratic Rep. Kathleen Rice told CNN outside the House chamber. "It's time for Nancy Pelosi to go, and the entire leadership team."

Rice attended the closed-door House Democratic caucus meeting on Wednesday morning but said she did not raise the issue in the session, and no other members brought up the idea of a leadership change. "I think that people were in shock" after hearing a report from Pelosi and the head of the House campaign arm that Democrats lost, but were doing better in contests in other GOP districts.

Massachusetts Democratic Rep. Seth Moulton echoed Rice's complaints in an interview with CNN's Jake Tapper. "This is certainly something that we have to discuss because it's clear that, I think, across the board in the Democratic Party we need new leadership. It's time for a new generation of leadership in the party."

Asked about her colleagues urging her to step down, Pelosi waved off the question and said she would address it at her weekly press conference on Thursday. But pressed if she had any plans to go anywhere, she replied "no."