Story highlights "I don't think it's divisive to go speak in a mosque. I'm surprised it took his eighth year to do it," Bush said

His comments come after Donald Trump and Marco Rubio criticized Obama for visiting a Baltimore mosque on Wednesday

Tilton, New Hampshire (CNN) Jeb Bush said Thursday it was "more than appropriate" for President Barack Obama to speak at a mosque, a departure from several of his Republican rivals.

But, Bush argued, it took too long for Obama to make his first visit to a U.S. mosque as president.

"I don't think it's divisive to go speak in a mosque. I'm surprised it took his eighth year to do it," the former Florida governor told reporters after stopping at a diner here. "The bigger issue is what have you done to deal with ISIS. It's important to recognize that peace-loving Muslims who are American citizens are as American as you and I are, but at the same time, this President has been derelict in his duties to destroy the caliphate."

His comments come after Donald Trump and Marco Rubio criticized Obama for visiting a Baltimore mosque on Wednesday , where he stressed a message of tolerance and condemned anti-Muslim rhetoric.

While Rubio acknowledged there's discrimination in America, he argued the bigger problem is radical Islam.

Read More