(CNN) Days have passed since President Donald Trump equated white supremacists and neo-Nazis with counterprotesters at a Charlottesville, Virginia rally that sparked violence over the weekend, the President's words are still rippling across the federal workforce at government agencies.

Career government employees tell CNN they're deeply disturbed by silence or what they perceive as weak statements, from the Cabinet members who lead their agencies.

There have been no public statements from Transportation Secretary Elaine Chao, a Chinese immigrant, or Treasury Secretary Steve Mnuchin, who is Jewish on where they stand on the President's rhetoric. Chao said the situation in Charlottesville " is hateful behavior and it is not who we are as Americans," but Chao stopped short of distancing herself from the President's remarks.

Both were physically standing behind the President as he argued Tuesday "there's blame on both sides," referring both to the white nationalists and those who demonstrated against their racism. Later, Chao was asked about criticism the President had lobbed at her husband, Senate Majority Leader Mitch McConnell, to which she replied "I stand by my man -- both of them."

CNN reached out to the spokesmen for multiple federal agencies. In some cases, it took officials more than a day to respond. In others, phone lines rang, and emails were sent without answer and cell phones went to voicemail.

Federal government employees told CNN it is imperative that they know where Cabinet members stand so they "know who they're working for."

"I'd like to hear from Administrator (Scott) Pruitt that he strongly denounces white supremacy and neo-Nazis," one career employee of the Environmental Protection Agency told CNN. "We watch Pruitt dismantling the regulations on polluters. We've assumed it is about corporate profit and his future political ambitions.

"But now, in light of the administration's effort to normalize white supremacy, Pruitt's deregulation efforts take on a whole new meaning," the official said.

Some at the Justice Department are "embarrassed" and "afraid," a career official at that agency told CNN. They fear Trump is "damaging people's perception of government workers," the official said, expressing concern about being seen as guilty by association.

Both employees asked that their names be withheld for fear of retaliation.

American Federation of Government Employees Union president J. David Cox echoed much of their comments. The union is currently celebrating "diversity week." AFGE is is the largest federal employee union representing 700,000 federal and DC government workers nationwide and overseas. "Cabinet members aren't pushing back on the President, saying 'you are wrong on this issue Mr. President,'" Cox said.

"Discrimination exists in this country. I've seen it and experienced it at all federal agencies. If the leadership at the top is not pushing back I believe it will trickle down to managers who will feel it is OK to treat people differently."

AFGE National Vice President Everett Kelley agrees. "It's not controversial to denounce bigotry."

The few Trump Cabinet officials who are speaking out

Ambassador to the UN Nikki Haley, born to Indian immigrants, has been silent on the President's defense of Confederate statues and his sympathy for the white supremacists who marched in Charlottesville. The US Mission to the UN told CNN they had no comment. The last time Haley spoke out on the topic was via Twitter on Saturday

"I know all to well the pain hate can cause. the American spirit that binds us has no place for actions like this. #prayersforcharlottesville."

I know all to well the pain hate can cause. The American Spirit that binds us has no place for actions like this. #PrayersForCharlottesville — Nikki Haley (@nikkihaley) August 12, 2017

As South Carolina's governor, Haley said the Confederate flag should not fly over South Carolina's state house, and the flag was removed

"Now I feel good because when my kids look up there won't be a flag and there will be one less thing to divide and it'll be one more reason for us to come together," she said at the time.

Housing and Urban Development Secretary Ben Carson, the only African-American Cabinet member in the Trump administration, came to the President's defense.

When speaking in Louisiana Monday, Carson called criticism of the President's rhetoric "little squabbles" that are being blown "out of proportion."

Carson defended the President. "Obviously when the President talks about the fact that hatred and bigotry are unacceptable, you know, he's talking about everybody. You think he would've said hatred and bigotry are unacceptable except by white supremacists and neo-Nazis? No, come on."

Education Secretary Betsy DeVos on Thursday shamed the white supremacists as "cowardly, hateful and just plain wrong," criticizing their views as "totally abhorrent to the American ideal."

She told department staff in a note, first reported by Politico and obtained by CNN, that they have "a role to play in rejecting views that pit one group of people against another."

The education secretary specifically called out the "white nationalists, neo-Nazis and other racist bigots" who gathered in the Virginia college town. She made no mention of controversial remarks made by Trump.

The secretary has previously been criticized for her handling of race and other discrimination-related issues. She called the colleges created to serve black students -- who were barred from then-white only schools -- as "pioneers" of school choice, and stumbled over questions about how she might handle private schools that receive federal funding and discriminate in admissions.

Five days after the Charlottesville march and multiple emailed requests later, Interior Secretary Ryan Zinke issued a statement to CNN.

"The racism, bigotry and hate perpetrated by violent white supremacist groups has no place in America. It does not represent what I spent 23 years defending in the United States military and what millions of people around the globe have died for," he said in the statement.

The statement added "We must respond to hate with love, unity and justice. I fully support President Trump and Attorney General Sessions in uniting our communities and prosecuting the criminals to the fullest extent of the law."

But AFGE president Cox is urging Cabinet members to go a step further.

"It's not enough to say Nazis are bad, they need to say whether they stand with or against the President's rhetoric. The President defended Nazi's. Do they stand with him on that?" Cox asked, adding at most agencies, leadership has not sent any email messages to the federal workforce stating where they stand.

"You can't waffle. Either discrimination is wrong or right. We can't have ambiguity. The workforce is very upset because they aren't hearing from their leaders. The leader sets the tone for subordinates."