"I'm optimistic about where we'll end up," former President Bush tells PEOPLE of the current political unrest in America

George W. Bush Breaks His Silence on the Direction of the Country Under President Trump: 'I Don't Like the Racism and Name-Calling'

For eight years since leaving the White House, George W. Bush has refused to criticize publicly those who succeeded him, saying he didn’t want to make an already tough job any harder for President Obama and, now, President Trump.

But one rocky month into Trump’s tenure, it’s harder to keep quiet.

Get push notifications with news, features and more.

“I don’t like the racism and I don’t like the name-calling and I don’t like the people feeling alienated,” Bush, 70, tells PEOPLE in an interview for the new issue of the magazine on newsstands Friday.

“Nobody likes that.”

Image zoom Peter Zambouros

As he discussed his most fervent passions in this post-White House chapter of his life—painting and helping the men and women he sent to war in Iraq and Afghanistan—Bush was reluctant to be drawn into conversation about the controversial Trump presidency.

Talking about the new president’s Jan. 20 inauguration, which both of the Bushes attended, the former president said, “First of all, we were thrilled to be there. It’s a beautiful ceremony, it really is. It’s a peaceful transfer of power, which is very reassuring.”

Image zoom Alex Wong/Getty

“Of course, we’ve been to very many inaugurals,” Laura added.

“It’s our sixth one,” said her husband.

Bush called the political climate in Trump’s Washington “pretty ugly” (“I’m not going back nowhere!” he added for emphasis), but said he isn’t feeling anxious about the direction of the country. “Not really. I’m optimistic about where we’ll end up. … We’ve been through these periods before and we’ve always had a way to come out of it. I’m more optimistic than some.”

Asked if he felt compelled to play a leadership role in these divisive times, Bush went on:

“No. When President Obama got elected, friends would call: ‘You must speak out! You must do this, you must do that.’ Turns out, other people are doing the same thing this time. I didn’t feel like speaking out before because I didn’t want to complicate the job and I’m not going to this time. However, at the Bush Center we are speaking up.”

The couple list some of the center’s work that stands in contrast to Trump’s isolationism: immigration ceremonies, women’s reproductive-health programs in Africa, and leadership training for Muslim women that the Bush Center brings to Texas from the Middle East. Asked if Trump’s determination to restrict immigration and travel from Muslim countries threatens the Bush Center programs, he shrugs. “Now that you mention it, it might bother me but we’ll figure out how to bring them over.”

“There’s a lot of ways to speak out,” the former president says, “but it’s really through actions defending the values important to Laura and me. … We’re a blessed nation, and we ought to help others.”

RELATED VIDEO: Click HERE to watch People Features: George W. Bush, Portraits of Courage

A spokesman for Bush previously told PEOPLE that neither he nor the former first lady voted for Trump – or for Democratic nominee Hillary Clinton – in November. Both of the Bushes, who had campaigned for Bush’s younger brother Jeb in the 2016 Republican primaries, cast write-in votes on Election Day for a candidate they did not reveal.

Bush says he has no line of communication with Trump, who has not reached out for advice, and notes that neither did Obama.

“It doesn’t hurt my feelings. I understand the nature of the job,” Bush says. “There’s a lot incoming when you’re the president and you just got to focus on your job.”

-with LINDSAY KIMBLE