John Kelly served less than two years as chief of staff in Donald Trump’s White House — and this week the man who also served as secretary of Homeland Security and was one of the armed forces’ longest-serving generals confirmed it was the toughest gig he ever had.

“[Being chief of staff] was an incredibly hard job; it was the least enjoyable job I’ve ever had,” he told an audience at Duke University on Wednesday. “But it was the most important thing I’ve ever done in my life.”

Kelly, who gave a talk titled “Leading America in a Time of Global Turbulence,” also notably rebuked his former boss’s much-touted claim that a wall along the U.S.-Mexico border would be necessary, or even effective.

“We don’t need a wall from sea to shining sea,” he said, adding that most Customs and Border Patrol agents would agree that a wall running the entire length of the Southern border would be “an awful waste of money.”

Trump’s correct in noting that immigration figures are increasing by some metrics. According to the Department of Justice, in Fiscal Year 2008, 42,710 immigrants filed for asylum — a legal protection status intended to help refugees fleeing unsafe conditions in their home countries. In 2018, that number surged to 161,005: a 275% increase in a decade’s time. But overall, there are substantially fewer border apprehensions than there used to be — the Fiscal Year 2017 rate was more than 80% lower than it was in Fiscal Year 2000, according to Customs and Border Patrol figures.

Further, most drug arrests happen at legal border checkpoints and not between them, where Trump’s new walls would be built.

“A small percentage of all heroin seized by [Customs and Border Protection agents] along the land border was between Ports of Entry,” the DEA wrote in 2018.

In addition to noting the expense of Trump’s proposed wall, Kelly also indicated to the audience Wednesday that Trump’s characterization of many immigrants as criminals or rapists was unfounded.

“They’re overwhelmingly not criminals,” he said, according to Fortune and other publications. “They’re people coming up here for economic purposes. I don’t blame them for that.”

Kelly’s claim is backed by some statistics: Undocumented immigrants were 16% less likely than native-born citizens to be convicted of murder in Texas in 2015, according to a case study by the CATO Institute, a libertarian think-tank.

Kelly’s blunt comments this week are also likely not the first time he has criticized Trump on his immigration policies.

According to a NBC report from last May, Kelly called Trump an “idiot” during a meeting about Deferred Action for Childhood Arrivals program — an Obama-era policy that allows undocumented children brought to the United States to stay and apply for work visas.

“He doesn’t even understand what DACA is. He’s an idiot,” Kelly said in one meeting, two officials who were present told NBC. “We’ve got to save him from himself.”

Though Kelly called the NBC report “total BS,” he ultimately left the position less than eight months later.

During his talk at Duke, Kelly also indicated that he might have also accepted the chief of staff job for former Secretary of State Hillary Clinton, had she won the 2016 presidential election and asked him to fill it.

“If Hillary Clinton had won the presidency, and she had called me and said, ‘I really need a good chief of staff here,’ I’d have probably done it,” Kelly said, according to the Hill. “Politics aside, it’s all about governing the country.”

While Kelly also told the Duke audience that Trump was “by no means a stupid man,” he also gave some impromptu advice to Mick Mulvaney, Trump’s acting chief of staff.

“Run for it,” he said, according to Politico.

Write to Abby Vesoulis at abby.vesoulis@time.com.