South Carolina Sen. Lindsey Graham, one of the 11 Republicans who sit on the Senate Judiciary Committee, made it clear Sunday that while he is willing to hear out Christine Blasey Ford about her sexual assault allegation against Supreme Court nominee Brett Kavanaugh, he has not heard enough evidence to "ruin Judge Kavanaugh's life over this."

"What am I supposed to do? Go ahead and ruin this guy's life based on an accusation?" Graham asked in an interview with "Fox News Sunday" host Chris Wallace. "I don't know when it happened, I don't know where it happened. And everybody named in regard to being there said it didn't happen. I'm just being honest. Unless there's something more, no I'm not going to ruin Judge Kavanaugh's life over this."

"But she should come forward, she should have her say. She will be respectfully treated," he added.

Ford alleges that Kavanaugh held her down, covered her mouth his hand and tried to remove her clothes after he and a friend locked her in a bedroom at a party in 1982, when they were all in high school.

Mark Judge, the friend who was allegedly in the room with Kavanaugh and Ford, denied having any recollection of the party. And in an email obtained by USA TODAY, a lawyer for a woman Ford also said was at the house said she didn't know Kavanaugh and that she did not remember the party.

More:Senate panel quietly investigating Kavanaugh assault claims amid testimony negotiations

Graham repeatedly expressed doubt about the allegation during the interview Sunday based on the amount of time that has passed since the alleged assault and the lack of evidence.

"This accusation has to be looked at in terms of our legal system, Graham said.

"Everything I know about Judge Kavanaugh goes against this allegation," he continued. "I want to listen to Dr. Ford. I feel sorry for her. I think she's being used here."

Wallace showed clips of Democratic Sens. Kirsten Gillibrand and Mazie Hirono criticizing Republicans treatment of Ford, but Graham wondered if "these two senators have an agenda that's related to their hatred of President Trump."

"I'm trying to be fair here and get this thing done in a reasonable way," he said.

Graham said that Judiciary Committee Chairman Chuck Grassley "has bent over backward to make this happen" and that the Iowa Republican had met six of 10 conditions set by Ford's lawyer for her to testify before the committee on Thursday morning.

He said Ford's team is still contesting the committee's insistence that only Ford and Kavanaugh testify, and that outside counsel be brought in to question them.

"They contest those two things," Graham said. "If they continue to contest those two things, there won't be a hearing."

"And that's it," he said, adding that if they couldn't agree to those terms that If they "means they really don't want to testify."

"Bring it forward, I will listen," Graham said. "But I'm not going to play a game here and tell you this will wipe out his entire life."

Wallace asked Graham if the Judiciary Committee Republicans, who are all male, wanted to bring in outside counsel to do the questioning because they feared "the optics" of them grilling Ford.

Graham explained that they wanted to bring in a lawyer because "we've got 11 politicians who haven't done a trial in about 20 years. I thought it'd be really smart to have somebody come in who knows what the hell they're doing, to ask the questions, to be respectful."