Washington (CNN) Republican lawmakers were still filtering into the ornate House Ways and Means Committee room and digging into a lunch of barbeque chicken, beef and pork tacos from a local food truck -- completely unaware of the bombshell that was about to drop.

As Kevin McCarthy got up to speak, there were audio problems with his microphone and some Republicans in the room couldn't hear him when he began talking.

He didn't directly announce he was withdrawing from the race, but McCarthy began talking about the need for House GOP members to figure out a way to unite.

The room went silent when they realized where he was going and members sat stunned when he ended his remarks saying, "I'm not the guy."

This account of one of the most dramatic days in the recent history of the House is based on interviews with dozens of lawmakers who witnessed McCarthy abandon his campaign for the speakership. His decision leaves the chamber adrift at a time when it faces crucial decisions in the coming months on whether the government will default on its debt or shut down entirely.

The day began with a tense meeting in the basement of the Capitol.

McCarthy was making his case when one of his colleagues took him to task for comments suggesting that the Benghazi oversight committee was aimed at hurting Hillary Clinton politically.

"It was a verbal blunder, and how can we have someone who made such a blunder and immediately make him our leader?" Rep. Dana Rohrabacher growled at McCarthy.

McCarthy apologized, but insisted he was "moving forward," attendees said.

The California Republican knew he could prevail in the midday House GOP conference election. But he was far more worried about the long slog until Oct. 29, when the full House would have to approve his nomination. That would require significant arm twisting -- and a number of more junior Republicans in conservative districts would have to take tough votes to preserve his chances at the speakership.

It was clear McCarthy was sensitive to that concern. Moreover, he had grown angry at a group of rabble-rousing conservatives in the House Freedom Caucus. He methodically courted and promised to work with them. He spoke to the conservative caucus Tuesday night and, by all accounts, he gave an effective presentation.

But the bloc still threatened to undermine his candidacy when it came for a vote on the House floor, people close to him said. He sensed he wouldn't be able to win them over unless he gave into a growing list of demands.

"I don't know what he could do," said Rep. Thomas Massie, R-Kentucky, when asked what moves McCarthy could make to win over the hard-right of his conference.

After the two-hour morning candidate forum Thursday morning, several of the younger House Republicans came to McCarthy and told him they supported him, but they were bracing for a firestorm when they went home for a week-long recess next week. They said there was growing pressure from outside groups against elevating someone who the right viewed as another John Boehner.

McCarthy recognized that if this much blowback was brewing now, it would only ramp up in the upcoming debate he would need to navigate on the debt ceiling, which needs to be raised by Nov. 5.

He made a decision he didn't want to put his party through more drama, and he wasn't going to be able to unite the bitterly divided GOP conference.

"Numbers, pure and simple. He had the votes to win the conference vote, but there just wasn't a path to 218," a source close to McCarthy.

McCarthy kept the matter mostly to himself, but told a handful of his closest allies ahead of time. He didn't tell Boehner until minutes before the meeting to elect the new speaker was about to start.

Since Conference Chair Cathy McMorris Rodgers was chairing the election, McCarthy also huddled with her briefly minutes before his announcement to tell her there was no need to hand out ballots: He was dropping out.