We know Donald Trump and his administration are doing everything they can to stop Muslims — law-abiding ones — from entering the country. They’ve tried a travel ban from Muslim-majority countries, they’ve tried putting a cap on legal immigrants, Trump is quick to condemn all Muslims for the actions of radical individuals in a way he never does when a terrorist is white, and he spent years talking about the importance of saying the words “radical Islamic terrorism” (or varieties on the phrase).

At the same time, a lot of prominent atheists (and allies) have also criticized Islam. Sam Harris once called it the “mother lode of bad ideas.” Ayaan Hirsi Ali and Maajid Nawaz have spent years trying to modernize the more conservative aspects of the faith — even to the point where they were labeled “anti-Muslim extremists.” And Richard Dawkins just got de-platformed from an event in Berkeley, California because a radio station because of his supposed “abusive” and “hurtful” words about Islam.

The question is: How much do those two groups overlap?

In a new video for Big Think, Dawkins emphatically denounces the actions of right-wing conservatives. His criticism is philosophical. He opposes radical interpretations of the Qur’an and the tragic consequences of dogmatic thinking. Trump, on the other hand, goes out of his way to make Muslims miserable.

It’s very unfortunate when you inadvertently find people agreeing with you who are the last people you would wish to agree with you. I mean, I despise Trump. I despise everything that he stands for. But it’s perfectly true that many people think that I ought to be on his side because he has these draconian, illiberal, horrible policies towards Muslims. I mean, trying to stop Muslims entering the country, what a horrible thing to do. What an impolite, unwise, illiberal, inhumane thing to do. And so I’m embarrassed if people on the Alt-Right agree with something that I say for the wrong reasons.

I get where he’s coming from. It’s frustrating when you find yourself agreeing with your ideological opponents on certain issues. It’s happened for me on the issue of controversial speakers on college campuses — I’m all for them if students want to bring them in, and there are smart ways to respond to speakers with bad ideas that don’t involve not letting them talk at all.

That doesn’t mean you agree with their motives or share other beliefs with them. It just means people who disagree on many things may find common ground elsewhere. The two lawyers who fought each other in Bush v. Gore notably joined forces in support of marriage equality. And that idea of coming together despite other differences is the whole basis of the interfaith movement.

So Dawkins is right to be frustrated with people who lump him with the alt-right camp because he criticizes Islam the same way he does other beliefs. Anyone who says it hasn’t looked into the issue deeply enough.