(CNN) There is a growing sense of concern inside the White House over the negative reaction to Matthew Whitaker being tapped as acting attorney general after Jeff Sessions' abrupt firing.

Whitaker, who was Sessions' chief of staff, has faced criticism since Wednesday afternoon's announcement for his previous comments on special counsel Robert Mueller's investigation.

Several senior officials told CNN they were surprised by the criticism, and believe it could potentially jeopardize Whitaker's chances of remaining in the post if it continues to dominate headlines.

Whitaker is expected to take over oversight of Mueller's investigation into Russian interference in the 2016 election and whether Trump campaign associates colluded with Russia. He has given no indication he believes he needs to step aside from overseeing the probe, according to one person familiar with his thinking, a belief echoed by White House officials. And a source close to the President told CNN that the idea of Whitaker ending or suppressing the Russia probe is not an option as of now.

But Whitaker has previously expressed deep skepticism about the probe, including arguing in a 2017 CNN op-ed that Mueller was "dangerously close to crossing" a red line following reports that the special counsel was looking into Trump's finances and calling Mueller's appointment "ridiculous" and "a little fishy" in a 2017 appearance on the "Rose Unplugged" radio program.

Read More