(CNN) He was there to survey the path of destruction left by Hurricane Maria. But when President Donald Trump visited Puerto Rico in October 2017 , the island's dire predicament was hardly the only topic on his mind.

People familiar with the visit said the President was distracted by other matters -- including his then-devolving war of words with North Korean leader Kim Jong Un -- as he toured devastated neighborhoods and took an aerial tour of the damage.

At one point, Trump pointed to the "nuclear football" -- a briefcase always in the President's vicinity that can be used to authorize a nuclear attack -- and claimed he could use it on Kim whenever he felt.

"This is what I have for Kim," he said, according to three people familiar who witnessed the remark.

The White House did not respond to a request for comment on the incident.

The episode came amid an increasingly acrimonious period that saw Trump boast of the size of his "nuclear button" and threaten to rain "fire and fury" on North Korea. Since then, he and Kim have developed a warm friendship and met for two summits.

But at the time, the casual reference to his nuclear capabilities was another sign of the spiraling rhetoric that marked his early interactions with Kim.

And, to some officials, it was an indication of Trump's disinterest in the plight of Puerto Ricans, who suffered for months without power and limited resources as their island recovered from the walloping storm.

"There were other topics that were being discussed and my view is that the sole focus of that trip should have been on Puerto Rico," said Puerto Rico Gov. Ricardo Rosselló in an exclusive interview on Thursday.

Rosselló, who declined to confirm Trump's remark about his nuclear football, said Puerto Rico was hardly the only topic at hand that day. But he also didn't deny the comments were made.

"He was talking about a whole host of other issues but I would rather leave those conversations internal," he said.

JUST WATCHED Puerto Rico Gov.: I'll punch the bully in the mouth Replay More Videos ... MUST WATCH Puerto Rico Gov.: I'll punch the bully in the mouth 02:03

Now, Rosselló hopes the President will return to the island, which is still struggling to recover fully from the hurricane. But Trump has shown few indications he's ready to travel there; instead, he's lambasted the federal dollars that have been allocated to the US territory and accused the island's government of misusing the funds.

In a closed-door meeting with Senate Republicans this week, Trump repeated those claims, insisting aid money be cut off to the island. Trump bemoaned the fact the territory was receiving more than $90 billion in aid, a figure that doesn't match what the island has actually received to date.

In the past, Trump has raised the prospect of money being diverted from disaster relief to pay for his proposed border wall. And he's threatened to reject $600 million in food stamps for Puerto Ricans.

Yet Trump claimed Thursday he's "taken better care of Puerto Rico than any man, ever," rebutting claims he's ignored the island's plight after Hurricane Maria.

"Puerto Rico has been taken care of better by Donald Trump than by any living human being and I think the people of Puerto Rico understand," he told reporters on the South Lawn.

The issue -- and the President's stated frustration with what the island has received up to this point -- is coming to a head as lawmakers work to reach a deal on a disaster relief package.

"The Trump administration is committed to the complete recovery of Puerto Rico," White House spokesman Judd Deere said in a statement this week. "The island has received unprecedented support and is on pace to receive tens of billions of dollars from taxpayers."

"However," Deere said, "the Trump administration will not put taxpayers on the hook to correct a decades old spending crisis that has left the island with deep-rooted economic problems."

Rosselló and other Puerto Rican officials chafe at the suggestion the hurricane recovery money would be used to correct the island's economic problems. Instead, they say Trump has been misguided by ideological aides, who see no political upside to providing assistance to an island with no electoral votes.

"I think he's getting bad information. That's why I invite him to the island to see his citizens. Puerto Ricans are proud citizens," Rosselló said. "It's just unfortunate that we have to have this battle of words when it's not the case in, say, Alabama or in Texas, where all of the resources were put at their disposition."

Trump's visit to Puerto Rico after the storm was largely limited to the areas around San Juan, which were not struck as hard as more remote locations. He did survey some of that destruction by air from his Marine One helicopter, but his tour of actual damage was limited to one neighborhood near the capital.

JUST WATCHED Anderson Cooper revisits this moment Replay More Videos ... MUST WATCH Anderson Cooper revisits this moment 03:23

Instead, he visited a church, where he was greeted by cheering crowds. He responded by tossing rolls of paper towels into the crowd like basketballs and boasting about the recovery efforts.

"Every death is a horror," Trump said in a briefing with local officials, including Rosselló. "But if you look at a real catastrophe like (Hurricane) Katrina, and you look at the tremendous hundreds and hundreds and hundreds of people that died, and you look at what happened here with, really, a storm that was just totally overpowering -- nobody has ever seen anything like this."

At that point, only 16 deaths related to Hurricane Maria were confirmed. A year later, a study put the death toll from the storm at 2,975 , a figure Trump disputed. Around 1,800 people perished from Hurricane Katrina.

Trump, Rosselló said, didn't fully grasp the storm's damage when he visited.

"I don't think he had the right perspective at that point in Puerto Rico," he said. "It was very early on during the process. The magnitude of the devastation was really magnified a little bit afterwards. But certainly a cause for concern after seeing some of the statements that were made."