Washington (CNN) Several employees at the Interior Department have told CNN that Secretary Ryan Zinke repeatedly says that he won't focus on diversity, an apparent talking point that has upset many people within the agency.

Three high-ranking Interior officials from three different divisions said that Zinke has made several comments with a similar theme, saying "diversity isn't important," or "I don't care about diversity," or "I don't really think that's important anymore."

Each time, Zinke followed with something along the lines of, "what's important is having the right person for the right job," or "I care about excellence, and I'm going to get the best people, and you'll find we have the most diverse group anyone's ever had," the sources said.

Interior last year unexpectedly reassigned 33 senior executive staffers, of which 15 were minorities, according to the lawyer of one of the staffers who was moved. Some of those who were reassigned have filed complaints with the US Merit Systems Board.

The accusations against Zinke come as he is under investigation by multiple agencies, including Interior's inspector general and Office of Special Counsel, regarding employee reassignment and taxpayer spending on possible politically related travel.

Interior Department spokeswoman Heather Swift vehemently denied that Zinke said anything along those lines of criticizing the need for diversity, saying, "the anonymous claims made against the secretary are untrue."

Swift added, "As a woman who has worked for him for a number of years in senior positions, I say without a doubt this claim is untrue, and I am hopeful that they are a result of a misunderstanding and not a deliberate mistruth."

Swift pointed to two women and an African-American who Zinke has appointed to senior leadership positions, and said "Zinke has filled several other senior positions at the career and appointed level with individuals from diverse backgrounds."

But Zinke's alleged comments were particularly surprising to those who feel the agency has struggled to recruit and retain a diverse workforce.

In a hallway meet-and-greet shortly after Zinke was confirmed, one staffer told CNN that Zinke was asked about diversity at Interior, a department with about 68,000 employees, of which more than 70 percent are white, according to the Office of Personnel Management.

"(Zinke) flat out said, 'I don't really think that's important anymore. We don't need to focus on that anymore.' He obviously needed someone to provide him with better talking points," the staffer said.

A similar comment was made during another hallway greeting session with a different group of employees.

"He said it several times. I think it's just how he speaks - he has his canned talking points," said the second source, who heard the same comment from Zinke months later at a holiday party.

A third person, someone who is a minority in a leadership position in the department, said he heard a similar comment during a management meeting.

"That told me everything I needed to know," the person said. "It's a hard business as it is, and then not to be respected or appreciated for the diverse perspective that you bring to the situation -- and that's why it's important in my opinion. It's the fact that we don't look at things the same way. When we have conversations about public lands and how they're used, we cannot afford to have a small percentage of people making those decisions."

In June, Zinke announced the unexpected reassignment of 33 senior executive staffers, angering some, including Joel Clement, a policy expert who says he was being forced to take an accounting assignment after he blew the whistle on climate change data in Alaska. He instead quit his job. He is fighting the reassignments, saying they are illegal, and the Inspector General and Office of Special Counsel are investigating. Neither investigating agency would comment.

Clement's attorney, Katie Atkinson, told CNN that she believes at least 15 of the 33 who were reassigned were minorities.

A CNN comparison with senior executive staff levels made available by the Office of Personnel Management show that a disproportionate number of those moved were racial minorities.

Only 28% of the 235 senior leaders at Interior self-report as minorities, but more than 40% of the 33 people who were moved in June without warning were non-white, based on the numbers provided by Atkinson.

Diversity has been a long-standing challenge at Interior. Seven of the 12 divisions have no minority leaders at the senior executive staff level, according to the latest OPM data, made available in September.

JUST WATCHED Zinke says 'Konnichiwa' after hearing WWII story Replay More Videos ... MUST WATCH Zinke says 'Konnichiwa' after hearing WWII story 02:35

"From the get-go we had heard from career executives at Interior that there was a feeling that there was a disproportionate impact on female and minority executives," said Jason Briefel, executive director of the Senior Executives Association, a professional group for federal career executives and leaders.

Briefel said he isn't privy to the department's decision-making process on the reassignments, and "maybe those folks happened to be the right person for the job," but he said he has heard the complaint frequently, and knows that some of those who were reassigned later filed complaints with the US Merit Systems Protection Board "to raise these claims."

The officials who spoke to CNN about Zinke's comments worry the reassignments may be a sign that what he said wasn't just a talking point.

"If you look at the actions he's taken, they are unbalanced in regards to minorities and women," said the minority manager who was upset by Zinke's diversity comment. "If you look at the people who were moved and you look at their race or gender, it's very obvious that this is a person that does not embrace the concept of diversity."

Zinke came under fire from the public and at least one member of Congress earlier this month over remarks seen by some as insensitive. In testimony before the House Natural Resources committee, he greeted Rep. Colleen Hanabusa (D-Hawaii) with the Japanese greeting "Konichiwa" after she told a story about her grandparents being held in internment camps during World War II, and asked why he was cutting funding to preserve those sites.

Days later, after numerous news stories calling the remarks inappropriate, Zinke doubled down in a comment to reporters, saying "How could ever saying 'Good morning' be bad?"