New York (CNN) President Donald Trump's fight to keep his tax returns private is at odds with his own thinking in 2013 and 2014 that releasing them as part of a presidential bid would make him look like a smart businessman who had spent years lowering his taxable income, according to two people with firsthand knowledge of conversations at the time.

Trump ultimately changed his mind after an adviser says he convinced him not to release his taxes, and he has spent years claiming he can't because he's under audit by the IRS.

Sam Nunberg, Trump's political adviser from 2011 to August 2015, tells CNN that during a meeting he had with Trump in the summer of 2013 at Trump Tower, the future president said he was comfortable releasing his tax returns and, even, that he thought it would be a good idea. Nunberg assumed this was because of how little Trump must pay in taxes.

"He thought he could defend the return," says Nunberg, who did not himself view Trump's returns. "I inferred from the conversation that he believed that it was a low number and he'd look savvy."

A second person, a former senior adviser to Trump, who also joined them for lunch that day, recalls Trump being enthusiastic about releasing his returns for this reason.

Nunberg remembers that at the time Trump had recently returned from delivering a political speech in Iowa and that his motivation to look like a scrappy businessman was fueled by the failed presidential bid of Mitt Romney. "He felt that Romney had avoided looking successful," says Nunberg. "Romney had posed beside a shopping cart in his jeans. Trump wanted to appear to be the opposite of that. He was proud of his business track record."

In May 2014, Trump told an Irish television station that he would "absolutely" release his tax returns if he entered the race. "If I decide to run for office, I'll produce my tax returns, absolutely," he said. "And I would love to do that."

It wasn't until November 2014 that Trump abandoned the idea, according to Nunberg. At that point it was still eight months before Trump announced for president. At another one-on-one meeting at Trump Tower, Nunberg says he convinced Trump to change tack, and told him that federal election rules obliged him to release only a broad financial statement, rather than his full tax returns. Trump liked the idea because he could show how rich he was, says Nunberg.

"He wanted to look rich rather than smart," Nunberg says.

Neither the White House, Trump's lawyer Jay Sekulow, nor the Trump Organization responded to a request for comment.

A losing fight

That change of heart nearly five years ago has had massive repercussions. During the 2016 campaign, Trump became the first major party nominee not to release his taxes in more than 30 years.

As President, he has faced numerous legal challenges seeking the release of his tax returns, including from House Democrats and the New York district attorney.

In fighting to keep them private, Trump has deployed an assortment of arguments both legal and prosaic, ranging from claims he's under audit by the IRS to simply stating his taxes are "none of your business." Trump has also acknowledged that he has fought to "very hard to pay as little tax as possible."

But after a string of court losses, Trump's unprecedented struggle to block the release of his tax returns is looking legally tenuous and appears more likely to head to the Supreme Court.

On Friday, Trump lost his appeal to stop House Democrats from subpoenaing his taxes from his longtime accountant Mazars USA. In a 2-1 ruling, the US Court of Appeals for the DC Circuit upheld a lower court ruling saying the firm must turn over eight years of accounting records.

Trump can appeal to the Supreme Court to stop Mazars, but courts, including the Supreme Court, previously have refused to curtail Congress' subpoena power.

A matter of vanity

Speculation has swirled around why Trump hasn't released his taxes, including that they could reveal long-denied ties to foreign interests or that he has donated embarrassingly little to charitable organizations. Trump's critics have also suggested that a full public airing of his tax records might prove that he has exaggerated his wealth and isn't as rich as he claims to be.

It's this last reason that is closest to the truth, according to Nunberg, who told CNN it's his impression that Trump's real motivation for not releasing his taxes was a simple matter of vanity.

A tax return of a New York real estate developer typically makes them look much less wealthy than they really are, on account of complex rules that include the ability for owners of profit-making buildings to write them off as losses.

Nunberg says the reason he suggested Trump not release his tax returns came down to three factors. First, by then Trump had told him that he was in fact under audit by the IRS. Secondly, he and Roger Stone, a mentor to Nunberg and himself a former Trump adviser, had started to realize that some of Trump's business history -- the bankruptcy of the Trump Organization in the 1990s in particular -- would come under attack and the returns might highlight that.

Third, Nunberg assumed, given his knowledge of the common tax practice of New York real estate magnates, there would likely be a vast discrepancy between Trump's net worth and what his tax returns showed -- and that this might be difficult to explain to voters in Iowa, New Hampshire and South Carolina. Nunberg knew that tax laws for commercial real estate developers are notoriously riddled with loopholes peculiar to that industry.

"I wanted him to run. I wanted him to feel as comfortable as he could. I didn't want any complications or hiccups. I tried so that this wouldn't hurt the Trump brand in any way," Nunberg said.

Changing his story

By early 2015, Trump was starting to slightly change the way he answered questions about his taxes. In February of that year, he told radio host Hugh Hewitt that he would "certainly show tax returns if it was necessary."

By October, he was hedging even more, telling ABC's George Stephanopoulos that he was considering releasing his tax returns. "I'm thinking about maybe when we find out the true story on Hillary's emails," he said of Democratic presidential candidate Hillary Clinton.

The financial statement Trump filed with the Federal Election Commission in July 2015 was 92 pages long and claimed $1.4 billion in assets and $265 million in liabilities.

Nunberg was fired from the campaign in August 2015, shortly after the financial statement was released.

During the presidential campaign Trump used the excuse of being under audit as the chief reason he could not release his taxes. He's repeated that defense as President. It's true that every president is audited every year, but there is no law that forbids them from releasing their returns while under audit.

"The President has fought releasing his tax returns since the early days of his campaign," said the former senior Trump adviser who says they still regularly speak with the President. "He has no interest in showing them or he would have released them. As usual I expect the Democrats will be disappointed if they are released, as they might just show Trump is a savvy, successful wealthy businessman."

But in May The New York Times reported that 10 years of Trump's tax records the paper had viewed, starting in 1985, appeared to show the exact opposite, and that Trump had lost $1.17 billion during that period.

The paper reported that, according to the tax records, Trump would have "lost" more money than any individual taxpayer in the entire country. Trump's attorney, Charles Harder, told the Times that statements about the records were "inaccurate" but pointed to no specific inaccuracies. He later added that IRS transcripts "are notoriously inaccurate."

In a response to the Times, a senior White House official said, "The president got massive depreciation and tax shelter because of large-scale construction and subsidized developments. That is why the president has always scoffed at the tax system and said you need to change the tax laws. You can make a large income and not have to pay large amount of taxes."

In other words, Nunberg's assumptions about why Trump's tax returns would be damaging to the Trump brand were spot on.