James Damore was fired from Google last week over his controversial 3,300 word essay on diversity. His memo put him in the good graces of the alt-right -- but he's now distancing himself from the movement.

"I do not support the alt-right," he told CNN Tech. "Just because someone supports me doesn't mean I support them."

Many alt-right personalities have expressed their support of Damore and his document, which criticized Google for its "politically correct monoculture" and critiqued its efforts to increase staff diversity. He had been photographed in his "Goolag" shirt by Peter Duke, a photographer who has been called the "Annie Leibovitz of the alt-right."

Damore's break from the alt-right comes after a white nationalist rally in Charlottesville, Virginia turned deadlly. Members of the alt-right, who have condemned the violence in Charlottesville, plan to protest outside Google offices in nine cities on Saturday. Damore said he's "likely not" going to participate.

Related: Google CEO cancels town hall due to leaks

Even as Damore clarified his personal political views, he argued adamantly that Silicon Valley is closed off to people it considers conservative.

"There's a very strong idea that the left ideology is the only ideology possible. We should be able to express differing opinions," Damore told CNN Tech. "I'm a centrist, and they're calling me a Nazi. That is a real problem."

Damore said he's not alone. He said some of his colleagues are afraid to express their ideologies but, privately, conveyed their full support of his memo.

"They literally say they agree with everything I'm saying. And that they don't feel they can bring their whole selves to Google," Damore said. He said the memo has divided many at the tech company.

"Hopefully it will show there has been a lot of political discrimination in the workplace and that needs to stop," he said, calling Google a "psychologically unsafe environment" because people feel as though they have to self-censor. "You have to stay in the closet and mask who you really are," he said.

Google CEO Sundar Pichai condemned parts of Damore's post that he said perpetuated stereotypes about women. Some experts have taken issue with Damore's arguments.

Related: Free speech on the job, and what that means

Damore's post claimed that women aren't suited for tech jobs for "biological" reasons. Men have a "higher drive for status," and women have higher rates of anxiety disorders -- making for "lower numbers of women in high stress jobs."

He said in the interview that he wasn't "saying anything about the women at Google."

"I'm saying that people that go into tech are interested in 'things' versus 'people' generally," Damore said. "As a population, there are fewer women that are interested in things versus people."

He said the memo was never meant to be leaked outside of Google -- and that if he could change anything, he'd eliminate the use of the word "neuroticism." "I would definitely change that because that has a lot of negative connotations," he said.