Story highlights He "of course denied it," Mike McCaul said. "I have no reason to not believe what McCarthy said."

McCarthy ended his bid to become House speaker Thursday

Washington (CNN) House Majority Leader Kevin McCarthy has repeatedly said his decision not to run for House speaker came down to math -- he couldn't get 218 votes after a bloc of conservative members said they would not support him. But he has also been forced to address tough questions about his personal life.

During a closed-door meeting with Texas Republicans last week when he was still a candidate for speaker, McCarthy was confronted "point blank" with a question about whether he carried on an affair with a fellow member of Congress, Rep. Mike McCaul told CNN. McCaul's account was confirmed by another GOP lawmaker who was in the room but asked not to be named because of the sensitivity of the conversations.

There is no evidence that the rumor is true.

He "of course denied it," McCaul said. "I have no reason to not believe what McCarthy said."

McCarthy on Thursday afternoon stunned Washington when he pulled out of the race for House speaker moments before a vote he was expected to win. Some conservative media outlets suggested the decision was connected to emailed threats from a conservative activist to expose a rumored affair between McCarthy and Rep. Renee Ellmers, R- North Carolina. CNN has not independently verified the email or its contents.

Read More