Washington (CNN) A new CNN poll conducted by SSRS finds that 70% of Americans are concerned about the number of employees who are serving in the White House without permanent security clearances, as the White House deals with the downgrade of clearances for dozens of employees who haven't yet received permanent approval for top secret access.

Further, 61% said President Donald Trump has done a poor job assembling a team of top advisers to work in the White House.

Concerns about staff without permanent clearances are highest among Democrats (85% are very or somewhat concerned), but independents (72% concerned) and Republicans (44% concerned) express high levels of worry as well. However, 77% of Republicans say they think Trump has done a good job assembling his advisers, something most Democrats (89%) and independents (63%) disagree with.

The poll was conducted before news broke that White House staffers operating without permanent security clearances -- including the President's son-in-law, who serves in a senior role at the White House -- would have their access to top secret information limited.

Trump's handling of that situation merited a 31% approval rating among those who had followed the news; 63% disapproved. Almost three-quarters overall (72%), including majorities across party lines, said those who have faced charges of domestic abuse should not be able to serve in government positions that require security clearances.

Rob Porter, a staff secretary in the White House, resigned after domestic abuse allegations surfaced, as did another White House staffer, David Sorensen. Porter and Sorensen have denied the accusations. The reports on the allegations raised questions about the White House's handling of security clearances, prompting Kelly's new policy and the limits imposed on Trump's son-in-law, Jared Kushner, and dozens of other White House officials.

Trump publicly defended Porter after the allegations became public. That support led some to question the President's attitude toward women, and all right on the heels of two stories about alleged extramarital affairs years before Trump took office.

Overall, 77% think it is true that Trump had extramarital affairs before becoming President. About two-thirds (65%) say it's true that a payment made by Trump's personal lawyer to an actress who stars in pornographic movies was to protect Trump's presidential campaign. And 56% believe it's true that the National Enquirer pays for exclusive rights to stories that may be damaging to Trump in order to keep them from being published.

Partisanship colors perception on each of these matters, with Democrats most apt to say they think each is true. Among Republicans, 52% think it is true that Trump had affairs before becoming President, but fewer believe the allegations around the payment to the actress (29%) and the National Enquirer (31%).

Overall, just 52% say Trump's personal life matters to them because the President's moral character is important, while 46% say they think it doesn't matter as long as he does a good job running the country. That is a reversal from public sentiment in February of 1999, when President Bill Clinton's personal life was in question. Back then, 53% said it didn't matter as long as he did a good job running the country, and 46% thought it was important.

Americans are also more apt than they were in 1999 to say that a president's extramarital affairs conducted before taking office should be relevant to how he is judged in office. About four in 10 say so now (38%), up from roughly one-quarter in 1999 (26%).

That shift rests on a sharp change by party. Back in 1999, when Clinton -- a Democrat -- was in office, just 10% of Democrats said such affairs were relevant to how a president is judged in office; now 61% say the same. Among Republicans, the 44% who considered them relevant in 1999 has fallen to 16% now, with a Republican in the White House.

White evangelicals, a key component of Trump's base in the 2016 election, express a bit more concern about these matters than do Republicans generally. They are more likely than Republicans generally to say that the president's personal life matters to them (44% among white evangelicals vs. 35% among Republicans generally) and more likely to say that affairs conducted before being elected are relevant for judging a president's time in office (26% among white evangelicals vs. 16% among Republicans generally).

Overall, 55% say Trump's moral behavior generally has hurt women in this country, more than think the same about his programs and policies (39%). Women are more apt than men to say Trump's programs and policies, as well as his moral behavior, have been harmful to women in the US.

The CNN Poll was conducted by SSRS on February 20-23 among a random national sample of 1,016 adults reached on landlines or cellphones by a live interviewer. Results for the full sample have a margin of sampling error of plus or minus 3.7 percentage points; it is larger for subgroups.