(CNN) Retired Adm. Mike Mullen, a former chairman of the Joint Chiefs of Staff under Presidents George W. Bush and Barack Obama, said Sunday on CNN's "Fareed Zakaria GPS" that he has been "extremely concerned" about military officials like retired Gen. John Kelly acting in political roles in President Donald Trump's administration.

Mullen said he worries that having military officials, "without being a politician, without running for office," acting politically could undermine the military as an apolitical institution.

Specifically, he said, he worries a "great deal" about Kelly "indirectly undermining us as a military" because of his role as White House chief of staff, a highly political job.

While many people have told him they have felt comfortable having Secretary of Defense James Mattis, Kelly, and former White House national security adviser H.R. McMaster in Trump's administration, all of whom have been generals, "I don't share that comfort," Mullen said.

Mullen said he's been to countries where the generals and admirals are in charge, which some of the public find comforting, and those are not countries Americans would want to grow up in.

Read More