Washington (CNN) House Speaker Paul Ryan described the message delivered to him by an octogenarian businessman a few months ago as akin to getting hit in the head "with a two-by-four, more or less."

He was in Springfield, Missouri, raising money for two House Republican colleagues when, as he recounts it, a "really successful guy" approached him to talk about family.

"Let me give you a piece of advice that I wish I'd listened to," the man told him, according to Ryan. "They used to say it's not the quantity of time you spend with your kids, it's the quality of time. That's a bunch of bull. It's both. And don't ever forget that. "

The idea that a public official would leave his position "to spend more time with family" and have that actually be the reason is an anomaly in Washington, one that draws knowing chuckles that imply a deeper, more dramatic rationale. But in a 30-minute interview with a small group of reporters in his Capitol Hill office, Ryan insisted that family -- to be specific, his three kids in or entering their teenage years -- really was the reason for the looming end of his 20-year congressional career, a decision made Sunday night at a family dinner in his hometown of Janesville, Wisconsin, and announced Wednesday.

The headwinds facing the Republican Party that have led most on Capitol Hill -- Republicans included -- to consider a future in the House minority the only likely end game? He's aware of them, but says that's not the reason he decided not to run again.

Frustration with President Donald Trump? He could do with fewer tweets, but their relationship remains in a good place.

Concern that he'd lose his race to a well-funded Democrat? Ryan is confident his seat will remain in Republican hands with or without him.

In truth, all of the above factored into the decision-making process (save for the last one -- Ryan and his team remained confident he was fine back home throughout). But the clincher that would put an end, for now at least, to the political career of a man who had become one of the faces of the Republican Party, was, in fact, his family.

The top House Republican appeared at ease as he reclined in a chair in his second-floor office overlooking the National Mall, hands clasped behind his head, reflecting on a career spent almost entirely pushing for his policy visions in federal government.

The successes in his mind were clear, most notably the passage of the 2017 tax law and the significant increase in military spending locked in by the recent $1.3 trillion federal spending bill. While he fell short in his goal of overhauling US entitlement programs, his ability to, in his words, "normalize" the discussion for significant changes -- something that in his earlier years would be considered radical and is now the mainstream of House Republican orthodoxy -- wasn't a small thing, either.

His biggest regret? The failure to repeal and replace Obamacare.

Does he have ideas about who should replace him atop the House Republican conference? Yes. But that will have to wait until after the election, he said.

His biggest concern? The rise of identity politics.

"It used to be a Saul Alinsky thing. The left was really good at it. Now the right does it," Ryan said, pointing to technology and social media as an accelerant. "That makes it hard just to have political goodwill in this country, because of all this polarization."

Asked how that concern squared with a President who has used a strategy that relies heavily on those same identity politics more effectively than anyone in recent political history -- a President who Ryan has publicly stood by throughout his time in the White House, the speaker exhaled audibly.

"Look, he won the election. He's the President of the United States. We're making really good progress on a lot of signature issues and we're making a positive difference in people's lives. I think if you can deny the oxygen of identity politics the best way to do that is to have a faster growing economy, more upward mobility, higher wages, getting people from poverty into the workforce. And those are some of the best tools to fight the seed corn of identity politics, the oxygen that feeds identity politics."

As far as Ryan's views on Trump -- from his initial refusal to back the candidate, which turned to hesitant support, which turned into at times strong public defense of the President -- they have been at the center of one of the more speculated about relationships on Capitol Hill.

In truth, according not just to Ryan but also to aides on both sides of Pennsylvania Avenue, the relationship between the two men has been solid. They talk constantly by phone -- often candid conversations, the speaker says.

One of the sharpest criticisms of Ryan -- not just from Democrats, but also from some Republicans -- is his perceived unwillingness to publicly attack Trump, even when he clearly disagrees on an issue or action, or, as is often the case, a tweet.

JUST WATCHED Paul Ryan: No 'devil's bargain' with Trump Replay More Videos ... MUST WATCH Paul Ryan: No 'devil's bargain' with Trump 01:44

"It works better to have private conversations than to have public disputes. It's more effective," Ryan says. "He appreciates that. He appreciates that when we have disputes or differences of opinion that we air them privately, and what I've learned is he learns. And he responds and he respects."

Asked if he could candidly advise against the idea of, say, firing Deputy Attorney General Rod Rosenstein or special counsel Robert Mueller -- something Trump has appeared to be teetering toward in recent days, Ryan starts to speak and then stops.

"I will keep our personal conversations personal, but I can say anything to him," he said. "We have very candid conversations with each other."

The other critique is that Ryan will deal with just about anything Trump does in order to get policy victories like the tax overhaul. Ryan doesn't necessarily reject that idea. Asked if there had ever been a point where drama or concern about the White House made the policy tradeoffs not worth it, he didn't hesitate.

"No, because I really believe the country was at a critical point," he said, then ticked through why he felt the tax overhaul and military spending increase were important.

JUST WATCHED Paul Ryan: I have no plans to run for anything Replay More Videos ... MUST WATCH Paul Ryan: I have no plans to run for anything 00:54

"That's what matters in posterity. The issue of the moment, the tweet of the moment, pales in comparison to the big policy changes that I really believe are going to make a big difference in people's lives, are going to move us in the right direction," Ryan said. "Those things completely overshadow the frustrations of the day or of the moment."

The frustrations of the day or moment will soon no longer be his problem. There are colleges to search for and some needed parenting relief for his wife, Janna, who Ryan estimates has shouldered about 90% of the parenting load during his time as speaker.

As to a future political career, Ryan made clear he's keenly aware that his national approval numbers might not be presidential material at the moment -- to say the least.

"That's what speaker of the House gets you," he said.

Asked flat out if he would run again for office, Ryan said no multiple times. He wouldn't be the first politician to go back on that pledge, but for the moment, all future plans have been moved off to 2019, he said. Except for maybe one.

"I wouldn't mind being ambassador to Ireland when I'm in my 60s," Ryan said with a laugh. "But that's about it."