(CNN) Ben Carson's campaign is in crisis -- and the finger-pointing is beginning.

Top players in Carson's orbit are bickering in public and his unorthodox campaign structure is proving to be more of a liability than an asset to the political neophyte. As the retired neurosurgeon tries to revive his flailing presidential bid, he might need to turn his campaign slogan -- heal, inspire, revive -- inward.

At the center of the fighting is Armstrong Williams, Carson's media-savvy business manager and longtime friend, who doesn't have an official role in the campaign, and Carson's official campaign leadership.

Williams has hit Carson's campaign team for putting the candidate in situations that have embarrassed him, such as with the recent Republican Jewish Coalition speech where Carson repeatedly mispronounced the name of Palestinian group Hamas.

"His campaign has to do a better job at preparing him," Williams said in an interview with CNN. "Don't give him speeches at the last minute. It's not only on him, it's on them."

Williams said he never wants to be sycophantic and he has often told Carson, "You have to be better than this," but his criticism often finds its way into the news cycle.

"I don't think he is always helpful," Barry Bennett, Carson's campaign manager, said of Williams' public criticism of Carson. "You always defend your candidate."

Doug Watts, Carson's communications director, said Williams' actions have rubbed the candidate the wrong way.

Photos: Who's running for president? Photos: Who's running for president? Ted Cruz, Donald Trump, John Kasich, Hillary Clinton and Bernie Sanders, Hide Caption 1 of 6 Photos: Who's running for president?



"So, ladies and gentlemen, I am officially running for president of the United States, and we are going to make our country great again," Trump told the crowd at his announcement. Businessman Donald Trump announced June 16 at his Trump Tower in New York City that he is seeking the Republican presidential nomination. This ends more than two decades of flirting with the idea of running for the White House."So, ladies and gentlemen, I am officially running for president of the United States, and we are going to make our country great again," Trump told the crowd at his announcement. Hide Caption 2 of 6 Photos: Who's running for president?



"These are all of our stories," Cruz told the audience at Liberty University in Virginia. "These are who we are as Americans. And yet for so many Americans, the promise of America seems more and more distant." Sen. Ted Cruz of Texas has made a name for himself in the Senate, solidifying his brand as a conservative firebrand willing to take on the GOP's establishment. He announced he was seeking the Republican presidential nomination in a speech on March 23."These are all of our stories," Cruz told the audience at Liberty University in Virginia. "These are who we are as Americans. And yet for so many Americans, the promise of America seems more and more distant." Hide Caption 3 of 6 Photos: Who's running for president? Ohio Gov. John Kasich joined the Republican field July 21 as he formally announced his White House bid.



"I am here to ask you for your prayers, for your support ... because I have decided to run for president of the United States," Kasich told his kickoff rally at the Ohio State University. Hide Caption 4 of 6 Photos: Who's running for president?



"Everyday Americans need a champion, and I want to be that champion -- so you can do more than just get by -- you can get ahead. And stay ahead," she said in her announcement video. "Because when families are strong, America is strong. So I'm hitting the road to earn your vote, because it's your time. And I hope you'll join me on this journey." Hillary Clinton launched her presidential bid on April 12 through a video message on social media. The former first lady, senator and secretary of state is considered the front-runner among possible Democratic candidates."Everyday Americans need a champion, and I want to be that champion -- so you can do more than just get by -- you can get ahead. And stay ahead," she said in her announcement video. "Because when families are strong, America is strong. So I'm hitting the road to earn your vote, because it's your time. And I hope you'll join me on this journey." Hide Caption 5 of 6 Photos: Who's running for president?



"This great nation and its government belong to all of the people and not to a handful of billionaires, their super PACs and their lobbyists," Sanders said at a rally in Vermont on May 26. Sen. Bernie Sanders , an independent from Vermont who caucuses with Democrats, announced his run in an email to supporters on April 30. He has said the United States needs a "political revolution" of working-class Americans to take back control of the government from billionaires."This great nation and its government belong to all of the people and not to a handful of billionaires, their super PACs and their lobbyists," Sanders said at a rally in Vermont on May 26. Hide Caption 6 of 6

"He does present us with some problems being on the outside of the campaign, Watts said. "Dr. Carson has expressed a little frustration here and there."

The infighting comes as Carson has fallen in the polls. He stood at 22% in mid-October, according to a CNN/ORC survey, dropping to a tie for third place at 14% in the same poll released last Friday.

A top fundraiser left earlier this month, citing a problem with Carson's leadership team. Poor communication led to an embarrassing New York Times story that raised questions about Carson's grasp of foreign policy. And a recent speech suggested that Carson, who as a young doctor obsessively prepared and studied, lacked fluency in international affairs.

Carson has dismissed the dip in polls, framing his decline as simply the nature of campaigns.

"Polls go up and down, when people were asking me when I was polling at the top spot was I excited, I said no, because polls go up and down, and I will say that now, I think you will see them continue to rise," he said on CNN in an interview with Jake Tapper. "At this stage of the game, very few people can sustain the kinds of attacks I have and still be at this level, so that tells you that there is a strong base of support there."

Carson is working to reverse the downward trend. He just released a plan for health care reform. He's traveling to Africa for a three-country tour. And his campaign just announced the addition of another foreign policy adviser.

For Carson, professionalizing his campaign will mean becoming a better candidate and learning how to assert himself more as the head of a sprawling operation that often misfires.

"He is a pretty easygoing guy and he does leave the campaign up to us and he is really very compliant and trusting of us," Watts said. "Sometimes we don't meet up to the standard."

Williams frames it differently.

"Dr. Carson is the leader, the campaign follows his lead, they are looking to him," he said. "Whatever happens, it will reflect on him."

'Poor Ben Carson'

Williams emerged this summer as the most visible spokesman for Carson, whom he has known for over two decades. As Carson climbed the top of the polls, Williams explained his appeal. And when Carson struggled after saying he would not back a Muslim-American president, Williams was there to double-down.

Photos: Ben Carson's career in politics Photos: Ben Carson's career in politics Ben Carson attends the National Action Network (NAN) national convention at the Sheraton New York Times Square Hotel on April 8, 2015, in New York City. Hide Caption 1 of 13 Photos: Ben Carson's career in politics Carson speaks during the 41st annual Conservative Political Action Conference (CPAC) at the Gaylord International Hotel and Conference Center on March 8, 2014, in National Harbor, Maryland. Hide Caption 2 of 13 Photos: Ben Carson's career in politics Carson speaks to guests at the Iowa Freedom Summit on January 24, 2015, in Des Moines, Iowa. Hide Caption 3 of 13 Photos: Ben Carson's career in politics Carson is surrounded by supporters as he waits to be interviewed at the annual Conservative Political Action Conference (CPAC) at National Harbor, Maryland, outside Washington on February 26, 2015. Hide Caption 4 of 13 Photos: Ben Carson's career in politics Carson speaks at the South Carolina Tea Party Coalition convention on January 18, 2015, in Myrtle Beach, South Carolina. A variety of conservative presidential hopefuls spoke at the gathering on the second day of a three-day event. Hide Caption 5 of 13 Photos: Ben Carson's career in politics Carson delivers the keynote address at the Wake Up America gala event on September 5, 2014, in Scottsdale, Arizona. Hide Caption 6 of 13 Photos: Ben Carson's career in politics Carson speaks during the 41st annual Conservative Political Action Conference (CPAC) at the Gaylord International Hotel and Conference Center on March 8, 2014, in National Harbor, Maryland. Hide Caption 7 of 13 Photos: Ben Carson's career in politics Carson speaks during the National Prayer Breakfast at the Washington Hilton on February 7, 2013, in Washington. Hide Caption 8 of 13 Photos: Ben Carson's career in politics Honoree and director of pediatric neurosurgery at Johns Hopkins University, Carson poses with actor James Pickens Jr. at the Jackie Robinson Foundation Annual Awards Dinner on March 16, 2009, in New York City. Hide Caption 9 of 13 Photos: Ben Carson's career in politics Before his jump into conservative politics, Carson was known for his work as a neurosurgeon. Carson was awarded the Presidential Medal of Freedom by then-President George W. Bush on June 19, 2008. At that time, he was the director of pediatric surgery at Johns Hopkins Hospital in Baltimore, Maryland. Hide Caption 10 of 13 Photos: Ben Carson's career in politics In a story that garnered international attention, Carson was ready to separate a pair of 10-year-old Indian girls, Saba and Farah Shakeel, who are joined at the head in New Delhi, India. Here, he addresses a press conference at the Indraprashtra Apollo Hospital on October 4, 2005. Hide Caption 11 of 13 Photos: Ben Carson's career in politics Carson and a team of 20 specialists approved the procedure after studying the girls' brains; however, their parents were worried about their daughters' lives and did not give doctors permission to operate. The surgery did not happen. Hide Caption 12 of 13 Photos: Ben Carson's career in politics Carson observes the start of neurosurgery proceedings at the Raffles Hospital in Singapore on July 6, 2003. Carson and Dr. Keith Goh, left, performed a complex operation that was unsuccessful to separate 29-year-old twins Ladan And Laleh Bijani, who were joined at the head. Hide Caption 13 of 13

He has served as a kind of "Carson whisperer," helping translate Carson's worldview and vouch for his personal story. But that closeness -- they speak several times a day -- has also posed problems.

Dogged by questions about his Detroit childhood, Carson lashed out at the media, yet Williams essentially told him to stop whining and learn to deal with the scrutiny that comes with a White House bid.

The not-ready-for-the-big-leagues problem with the campaign was most recently on display at the Republican Jewish Coalition speech.

Carson, who normally speaks extemporaneously, spoke from a prepared text, rarely looking up at his audience. When he seemed to pronounce Hamas, as "hummus" at times, it launched a barrage of criticism and outright mockery.

Bennett brushed aside the impact of that appearance, saying that "less than 1% of America watched anything that had to do with the RJC speech."

But those watching were Republican insiders who were looking for reasons to either dismiss Carson or give him another look.

"Poor Ben Carson. Someone should have told him how to pronounce Hamas," tweeted Ari Fleischer, former press secretary to President George W. Bush. "He sounds like he's not familiar with the group."

Poor Ben Carson. Someone should have told him how to pronounce Hamas. He sounds like he's not familiar with the group. — Ari Fleischer (@AriFleischer) December 3, 2015

Watts acknowledged it wasn't the campaign's best moment.

"He was well aware of the speech and the content and the development of the speech," Watts said. "But clearly he could have used more time reading it over several times, given that he rarely delivers a prepared text. I will take responsibility for that."

New York Times story

Williams has also fumbled in high-profile ways, botching a recent New York Times piece about Carson's foreign policy by giving a reporter the name of Duane R. Clarridge, an infrequent adviser who then trashed Carson, implying the candidate doesn't grasp the intricacies of world politics.

"I had no idea that Armstrong was talking to the Times," Bennett said. "I had never heard of that guy."

The campaign was completely blindsided by the story, later going on cleanup duty, which entailed Carson brushing back Williams.

"He has nothing to do with the campaign. Nothing," Carson told reporters in Mobile, Alabama, last month , shortly after the story ran. "Armstrong can comment on his own behalf. He does not speak for the campaign at all. He does not speak for me. He speaks for himself."

Careful to note how difficult it is to run a campaign at the presidential level when the candidate is a novice, Williams said he gets along with everyone in the campaign. He said Watts recently sent him an e-mail saying he was thankful Williams often had his back.

"I prefer to give people who run his operation as much leeway as possible and get out of their way," Williams said. "I respect them and I want them to do their jobs without hanging over their heads."

But he also suggested he could help right the ship.

"If I travel with him, there will be no gaffes if there is a press conference," Williams said, adding that Carson's press secretary Deana Bass would likely take cues from how he ran a recent press event in South Carolina.

Yet Williams acknowledges he hasn't always been helpful as he tries to balance his role as a surrogate and tough-love friend.

"Chris Wallace, that hurt him, and then The New York Times and myself saying he was on a learning curve, that hurt him," Williams said. "If I could do it over again, I would."

Carson's future

Bennett said they "are operating a single campaign at this point." A heavy focus on Iowa, where a top-three finish is a must, will bring Carson to a state he hasn't had a steady presence in. The campaign boasts a heavy social media footprint, with Carson bypassing the traditional press to connect directly with his base; Carson has 5 million fans on Facebook. Bennett is focused on a pool of 103,000 people.

"If we can turn out 35,000 to go to caucus, we can win," he said.

But turning out voters in Iowa takes discipline and organization, things the campaign and candidate have struggled with.

"It's a growth process for all of us," Williams said. "We are learning, we have been reflective to see what we need to do. You will see the difference."

On Friday morning Williams sought to downplay problems with the campaign.