Washington (CNN) As President Donald Trump and Vice President Mike Pence prepared to gather for their weekly lunch in August 2017, the President told his staff to add two more plates.

Both men had just welcomed new chiefs of staff -- retired Marine Corps Gen. John Kelly and Nick Ayers, a then-34-year-old Republican political consultant from Georgia -- and Trump decided to wave the pair into his private dining room off the Oval Office.

Until then, a Cabinet member would occasionally join them, but the meals were largely a chance for Trump and Pence to spend time together alone, chatting about politics, policy and whatever popped into Trump's mind -- sometimes prompted by the television in the room tuned to Fox News.

But in August 2017, the lunch went from a regular tête-à-tête to a four-man affair, one that became a more formal opportunity for the two offices to coordinate on strategy, policy and scheduling. For Ayers, Pence's new chief of staff, they were useful in another, perhaps more important way: he now had regular face-time with the President. With each passing lunch, Trump grew to know and like Ayers more, two sources close to the President said, allowing Ayers to build a strong personal rapport that led Trump to seek Ayers out as his next chief of staff.

Trump announced on Saturday that Kelly would leave his post at the end of the year and sought to iron out a plan for Ayers to succeed him. But it was Ayers who declined to take the position, refusing the President's requirement that he commit to a two-year tenure as chief of staff.

Instead, Ayers announced he would be leaving the White House entirely at the end of the year. He is expected to run Trump's America First super PAC, a source familiar with the matter said.

"Thank you @realDonaldTrump, @VP, and my great colleagues for the honor to serve our Nation at The White House. I will be departing at the end of the year but will work with the #MAGA team to advance the cause. 🇺🇸#Georgia," Ayers wrote in a tweet.

Thank you @realDonaldTrump, @VP, and my great colleagues for the honor to serve our Nation at The White House. I will be departing at the end of the year but will work with the #MAGA team to advance the cause. 🇺🇸 #Georgia — Nick Ayers (@nick_ayers) December 9, 2018

Youth and ambition

Interviews with nearly two dozen current and former White House officials, former Ayers colleagues, sources close to the President and Republican congressional staffers portray an ambitious aide who has worked to insulate his current boss from the chaos of the West Wing, while also angling for a bigger job that would place him squarely in the middle of it.

Trump's relationship with his chief of staff has reached a stalemate and is no longer seen as tenable by either party as the two have stopped speaking in recent days. Trump has begun to envy the smoothly operating vice president's office, which Ayers has managed to keep distanced from the daily scrum and scandal of the White House. Ayers has cultivated key allies, including Ivanka Trump and Jared Kushner. He also boasts an impressive track record in Republican politics that could serve the President well in the run-up to his 2020 re-election.

But Ayers' meteoric rise has also earned him his fair share of critics, including a few inside the White House. While plans were floated in November for Ayers to become the new chief of staff, multiple sources told CNN, they stalled amid the President's reluctance to fire Kelly -- who typically does the firing for Trump -- and the backbiting Ayers has faced from some of his West Wing colleagues.

Several of Trump's top advisers have voiced concerns to him about Ayers, with some threatening to quit if he was tapped for the job. One of Ayers' top West Wing detractors during the process was Kellyanne Conway, the combative counselor to the President who vehemently opposed Ayers' hire as Pence's chief of staff last year, two former White House officials and a source familiar with the matter said.

Conway disputes those allegations, telling CNN: "I have zero beef with Nick Ayers."

Outside the White House, former colleagues of Ayers say his relative youth and outsized ego -- conspicuous even in a world known for naked ambition and self-aggrandizement -- have rubbed fellow political operatives the wrong way. His allies say that people are just jealous or insecure.

"I think every job he's ever had he's been one of the youngest people to ever have it. And I think that's threatening to some people," said Alex Conant, who worked with Ayers on former Minnesota Gov. Tim Pawlenty's 2012 presidential campaign.

Though he's only 36, Ayers has amassed a small fortune that, according to recent financial disclosures, is between $12 million and $54 million. That's been built up through financial investments, fees generated by his own political consulting firm and his former role as a principal in an ad-buying firm called Target Enterprises, which has served as the media buyer on nearly every race Ayers has worked on since he joined in 2011.

The arrangement allowed Ayers to earn a consultant's salary while also influencing campaign spending in a way that benefited him financially, a practice that is not illegal but has raised consternation among fellow consultants. A source familiar with the matter insisted all of the candidates Ayers has serviced were aware of the financial arrangement behind his consulting.

Still, his finances and involvement with political dark money groups could become political baggage down the road. One of them, Freedom Frontier, for whom he consulted, is the subject of two recent ethics complaints, the most recent of which was filed with the IRS on Tuesday. That complaint, filed by Citizens for Responsibility and Ethics in Washington, contends the group's political spending exceeded legal limits in violation of campaign finance laws.

Efforts to reach officials with Freedom Frontier for comment were unsuccessful.

The prodigy

JUST WATCHED Who is potential Kelly replacement Nick Ayers? Replay More Videos ... MUST WATCH Who is potential Kelly replacement Nick Ayers? 02:21

Ayers was just 19 when he dropped out of Kennesaw State University in Georgia to be Republican gubernatorial candidate Sonny Perdue's body man. Ayers quickly became one of Perdue's most trusted advisers and helped guide the campaign to victory, making Perdue the first Republican governor in Georgia since Reconstruction. Over the next few years, Perdue tapped Ayers to manage his first re-election campaign and later to lead the Republican Governors Association as executive director.

As head of the RGA, Ayers was credited with streamlining the organization and dramatically expanding its fundraising prowess. Though he wasn't even 30, Ayers' work caught the eye of Pawlenty, who hired him to manage his short-lived 2012 presidential campaign that ended before the primaries even began that year.

"I like to joke that it was more brief than a Kardashian marriage," Pawlenty told CNN. "My brief and ill-fated campaign for president failed for a variety of reasons, but none of them related to Nick ... He's a really, sort of prodigy-level talent."

Still, Ayers faced criticism from some of his former colleagues over Pawlenty's failed campaign. Also that year, a video surfaced of his arrest on charges of drunk driving in 2006, when he was working for then-Gov. Perdue's re-election campaign. In the video, Ayers can be heard trying to engage the arresting officer in a conversation about politics, and implying how badly the incident would reflect on the governor's race. The charges were eventually reduced to reckless driving.

Ayers subsequently called the incident a "maturing moment" in a 2010 Washington Post article

Tension with Conway

Ayers' journey into the Trump fold began in 2011 when he first met then-Rep. Pence in Washington. Though he and Pence didn't work together officially, they stayed in touch. In 2015, Ayers joined Pence's gubernatorial re-election campaign as a general consultant. At the time, Conway was the campaign's pollster.

Three people familiar with the campaign said Ayers was critical of Conway's polling and wanted to oust her from the role. A fourth person familiar with the time period refuted that claim.

The tables turned two years later, when White House officials mulled enlisting Ayers to take over as Pence's chief of staff in the spring of 2017 and Conway stepped in to try and block the hire, two former White House officials and a source familiar with the matter said. In one incident, two sources said she erupted at then-White House chief strategist Steve Bannon, who was pushing for the move.

Conway denied having ever opposed Ayers -- then or now -- and criticized the "endless speculation" in the media.

"I preceded Nick on Gov. Pence's consulting team and I preceded Nick in this White House, but I was very happy when he followed and joined in the fun on both times on both occasions," Conway said. "I am pro-John Kelly and pro-Nick Ayers. As somebody who deals routinely and directly with the President and the vice president, I have an excellent working relationship with their excellent chiefs of staff. As far as I know, neither of those jobs are available."

Ayers declined to comment for this story, but Alyssa Farah, the vice president's press secretary, dismissed the reporting as "absurd and it's false" and maintained that Ayers and Conway are "personal friends."

Relationship with Trump

After Pence brought him in as chief of staff, Ayers quickly expanded his portfolio beyond the day-to-day operations of the vice president's office. That included setting himself up as a key liaison to the Oval Office and offering to help in times of chaos.

"When stuff was going crazy or there would be some bad story and we'd be in crisis mode, Josh (Pitcock, Pence's former chief of staff) wouldn't come in and be like, 'How can I help?'" one former White House official said. "Nick would."

A senior administration official rejected the notion that Pitcock wasn't as engaged as Ayers during his time as chief of staff, telling CNN that Pitcock "did his fair share of crisis management during his tenure as chief of staff."

Ayers also made a point of steering Pence away from the chaos of the West Wing. A senior White House official said Ayers determined a narrow set of targets for the vice president and his team to accomplish and built out a plan to focus almost exclusively on those goals. The official said Ayers did a better job of coordinating between the President and vice president's office -- the weekly lunches being a prime example.

Ayers also worked to improve his own access to the President, multiple former Trump campaign officials told CNN. "He wanted to be around Trump a lot and the family. He made sure he was real visible," one former Trump campaign official said.

A particularly important moment in the relationship between Ayers and the President came in January, when Ayers invited Trump to sit in his private box at the college football playoff title game between the Alabama Crimson Tide and Ayers' home-state team, the Georgia Bulldogs. A photo that Ayers posted on Twitter shows the two men standing side by side in the stadium box, smiling and wearing identical red ties and navy suits. Like Trump's, Ayers' tie hangs on the long side, just past his belt buckle.

A political chief of staff?

By the end of this summer, as the midterms approached and the emphasis turned toward electoral politics, Ayers became more visible in the West Wing.

He played a key part during White House discussions about political strategy, the candidates Trump should endorse and where he should travel. Ayers' political insights only served to amplify something several of the President's friends and advisers have voiced to him, which is that Kelly lacks the political know-how with which most White House chiefs of staff are armed.

And as the President begins to focus on his 2020 re-election campaign, those same friends and advisers are urging him to replace Kelly with a politically savvy successor.

"From now until November 2020, every decision that should be made in that building should be a political one. Every decision should have an eye on the re-election," one source close to the President said. "Nick understands that."

Though Ayers is now out of the running, whoever takes the chief of staff job will still be expected to usher in a dramatic shift in tone, style and expertise.

Kelly has no experience running campaigns and shies away from interacting with the donor class and Trump's billionaire friends -- like when he was spotted off to the side with one of his aides while Trump entertained those friends on election night.

Like Kelly, though, the next chief of staff will be pressed to manage White House infighting, a mercurial President and the free-flowing discourse between Trump and his kitchen cabinet of outside advisers.

The challenges of the coming year could eclipse those Kelly has faced in his 17 months on the job.

Trump will have to contend with the new reality of a divided Congress and a feisty Democratic House eager to bring the full force of its oversight power down on Trump and his administration. Special counsel Robert Mueller could also soon complete his lengthy investigation into allegations of Russian collusion and questions of obstruction of justice, a prospect that has unsettled the President.

Finances

It's possible that Ayers' past political work and issues over his finances could have become an issue if he had taken the White House chief of staff job. Ayers did not immediately disentangle himself from his businesses upon joining the administration, selling his consulting firm, C5 Creative Consulting, nine months later in April to Phil Cox, a fellow Republican strategist and former colleague who is close to Ayers.

Some White House aides have questioned Ayers' motives in the delay and say that his consulting business is at odds with Trump's "drain the swamp" slogan. A person familiar with the matter, who requested anonymity to discuss the confidential process, said Ayers was "very willing from the beginning to do whatever was required" to be in compliance with government ethics regulations. The source said it took time for Ayers to be able to sell the firm at fair market value.

"He is not an ethically challenged person," the person said. "Those folks exist in the Trump administration and Nick is not one of them."

A review of Ayers' financial disclosure form from October 2017 shows the complexity of his assets and holdings. Ayers listed 31 sources of income that exceeded $5,000 a year each, with some of those including consulting for Aflac, Coca-Cola, and political campaigns like Pence's gubernatorial re-election campaign in Indiana and Eric Greitens' successful bid to become governor of Missouri in 2016. Ayers also owns stock in various companies and agricultural real estate for forestry and pecans in Georgia.

Loyalty

Despite hailing from more traditional Republican circles, Ayers has sought to firmly establish himself as a Trump loyalist, an important marker of trustworthiness for the President. It doesn't hurt that before joining the administration in 2017, Ayers co-founded America First Policies, which has become the principal outside political group supporting the President and is allied with the super PAC Ayers will now be heading.

At times, though, Ayers' political tendencies have gotten him in trouble, including over an August 2017 New York Times report that Ayers told GOP donors that Pence wanted to be ready to run for President in 2020 should Trump be unable to run for re-election. He also drew flak in April over a plan to install Jon Lerner, a Republican pollster to whom he is close, as the vice president's national security adviser. Lerner was publicly forced to withdraw after the appointment caused tensions in the administration, with some reports signaling Trump was upset after learning of Lerner's association with a Republican political group that savaged him during the 2016 campaign. Lerner and Ayers are both close with outgoing UN Ambassador Nikki Haley.

But Ayers' loyalty to Trump has largely shined through. During a closed-door GOP fundraiser last year, Ayers leaned into the President's willingness to knock fellow advisers and pulled no punches in criticizing Republican lawmakers who he described as insufficiently supportive of the President.

"Just imagine the possibilities of what can happen if our entire party unifies behind him? If — and this sounds crass — we can purge the handful of people who continue to work to defeat him," he told the donors.

The comments rubbed lawmakers on Capitol Hill the wrong way. Two senior Republican aides told CNN that Ayers' had "pissed a lot of people off," and that lawmakers were taken aback since Ayers was so new to the White House.

"People were like, 'Who the hell is this guy?'" one of the aides said.

This story has been updated to include additional comment on Pitcock.