Story highlights Perry source in South Carolina: "We've been told to hold until we hear from Austin"

"There was nobody who had worse debates than he did," Carville says

Earlier, Perry had vowed to carry on campaigning regardless of Iowa finish

He saw his numbers fall in recent weeks

Following a disappointing projected fifth place finish in the Iowa caucuses, Texas Gov. Rick Perry said he'll return to Texas on Wednesday to reassess his candidacy.

"With the voters' decision tonight in Iowa, I decided to return to Texas, assess the results of tonight's caucus, determine whether there is a path forward for myself in this race," he told supporters late Tuesday night at his Iowa headquarters in West Des Moines.

Earlier Tuesday, Perry had vowed to carry on campaigning, if he finished poorly in Iowa, saying that results from the nation's first Republican nominating contest wouldn't be the be-all end-all in the presidential election.

"We have been told to hold until we hear from Austin," said a Perry source in South Carolina, where the candidate was preparing to launch an aggressive statewide bus tour on Wednesday. "It looks like he is not coming this week. He has definitely canceled his plans."

Perry's caucus concession speech Tuesday night had some wondering if this were the beginning of the end for the one-time front-runner.

"With a little prayer and reflection, I'm going to decide the best path forward," Perry said, as his family stood beside him.

JUST WATCHED Perry: Long way from being over Replay More Videos ... MUST WATCH Perry: Long way from being over 04:29

JUST WATCHED Perry on caucus day: 'It's game on' Replay More Videos ... MUST WATCH Perry on caucus day: 'It's game on' 01:15

CNN political contributor Ari Fleischer, the former White House press secretary under President George W. Bush, said it may be possible that Perry will bow out of the presidential race altogether.

"That is what will be fascinating if Perry drops out," said Fleischer. "If Perry drops out, if Bachmann drops out, it will be a four-person race."

Perry surged to the top of the GOP presidential pack when he jumped into the race mid-August but quickly saw his numbers fall in the polls after a series of uneven debate performances and gaffes.

"He was on fire for a while," said CNN contributor and Democratic strategist James Carville. "There was nobody who had worse debates than he did. He started as a major candidate. We were ready to be blown away. But I am sorry, he was the worse ever. Rick Perry is the worst presidential candidate in American history."

The biggest gaffe was Perry's infamous "brain freeze" in early November during the Republican presidential debate in Rochester, Michigan.

Perry's one-time promising presidential run turned into a punchline after he struggled to name the third of three federal agencies he would cut if elected president.

"I will tell you, it is three agencies of government when I get there that are gone," Perry said. "Commerce, Education, and the -- what's the third one there? Let's see. OK. Commerce, Education, and the ... "

"EPA?" Mitt Romney offered.

"EPA. There you go," Perry said.

When pressed by moderator John Harwood if the Environmental Protection Agency was indeed the third agency he proposed shuttering, Perry admitted that it wasn't. He then attempted again to remember the details from his plan.

"The third agency of government I would -- I would do away with Education, the Commerce, and, let's see. I can't. The third one, I can't. Sorry. Oops."

Several minutes later, Perry clarified that the agency he was trying to remember was the Department of Energy.

The next day he went on a media blitz trying to laugh away the mistake and later had a self-effacing appearance on David Letterman's "Top Ten List."

Some of Perry's quips were:"You try concentrating with Mitt Romney smiling at you. That is one handsome dude! I had a 5-hour Energy drink six hours before the debate."

But the damage was done.

Also in November, Perry said the voting age in the United States is 21 instead of 18 and that the 2012 presidential election will take place on November 12 instead of November 6.

Days later, during an interview with Fox News, Perry incorrectly identified the New Hampshire state's early voting contest as "caucuses" instead of a "primary."

Next week, the Republican candidates head to New Hampshire for the nation's first primary on January 10.

With his announcement Tuesday night, it is unclear if Perry will be among them.