Washington (CNN) House Democratic leaders and rank-and-file members are dismissing calls to impeach Supreme Court Justice Brett Kavanaugh, with some arguing the House has limited investigative resources and others saying it is a politically toxic issue.

Asked on Tuesday night if she sees the House spending any time on the Kavanaugh matter, House Speaker Nancy Pelosi responded with a simple "no."

Attention turned to Kavanaugh after a book published this week included a previously unreported incident of alleged sexual misconduct by Kavanaugh while a student at Yale. CNN is not reporting details of the accusation and has not independently corroborated the account. The female victim declined to be interviewed by the book's authors, and her friends told the authors she does not recall the incident. Kavanaugh has denied all allegations against him.

Since The New York Times published details from the book on Sunday, a number of Democratic presidential candidates and lawmakers have come out in favor of impeaching Kavanaugh. But House Democrats aren't eager to act.

Rep. Katie Hill, a California freshman, said she believes talk of impeachment could hurt Democrats on the campaign trail.

"I do think there need to be questions asked about why the FBI had this information and it wasn't shared," she said. "But I will say also that this is one of the most divisive things that happened during the course of the campaign. As a sexual assault survivor, I really could not understand why it was so divisive. ... I'm in one of the more liberal leaning swing districts, and still my poll numbers just dropped dramatically because I was so outspoken against him."

"No matter what we believe, fervently, to the bottom of our souls, on whether he should be there or not, I know that this is not the issue that we want to be talking about just in terms of election," she added. "I would like us to be able to just do what's right, but I know that for many of my colleagues, it's a really, really tough one."

Pennsylvania Rep. Matt Cartwright also said his party should not prioritize the issue.

"Democrats are focused on the bread and butter, kitchen table issues facing ordinary Americans," he told CNN. "I'm not saying proper oversight should not be conducted. It should be and it must be. But if you're asking me about priorities, it's taking care of Americans and their everyday issues."

And Rep. Jamie Raskin, a member of the Oversight and the Judiciary committees, said that Democrats should seek more information, "but it's not at the top of our agenda right now."

"Dealing with the Trump administration is like a video game where all of the monsters are flying at you 60 miles an hour. And so people have to deal with them as they come, and we have to do our best to sort it out and prioritize our fact-finding investigations," he told reporters.

"We're in the middle of looking at a whole set of unprecedented lawless activities by the White House. And it's not that this is disconnected from it," he said of Kavanaugh. "It's relevant to it. I mean, judicial corruption contaminates everything. So this is relevant and we should get the information that we can, but we also have limited resources."