Bridgewater, New Jersey (CNN) Two senior Trump advisers -- one inside the White House and another who recently departed -- signaled Sunday that the knives are out for Steve Bannon, President Donald Trump's controversial chief strategist.

The comments come as a source inside the White House tells CNN that White House chief of staff John Kelly has soured on Bannon, a political operative with deep ties to the 'alt-right' and the former head of the conservative news site Breitbart.

Bannon is seen as pursuing his own agenda, which does not mesh with the power structure Kelly is putting in place, the source added.

National security adviser H.R. McMaster was asked three times by NBC's Chuck Todd on Sunday whether he can work with Bannon in the White House. McMaster dodged the question each time and eventually left it an open question as to whether Bannon is motived by advancing the President's agenda.

"Can you and Steve Bannon still work together," Todd asked.

Read More