We combine community‐level outcomes of 27 votes about immigration issues in Switzerland with census data to estimate the effect of immigration on natives' attitudes towards immigrants. We apply an instrumental variable approach to take potentially endogenous locational choices into account, and we categorize immigrants into two groups according to the cultural values and beliefs of their country of origin to understand how the cultural distance between natives and immigrants affects this relationship. We find that the share of culturally different immigrants is a significant and sizable determinant of anti‐immigration votes, while the presence of culturally similar immigrants does not affect natives' voting behavior at all in most specifications. We argue that the differential impact of the two groups of immigrants is, at least in part, driven by natives' concerns about compositional amenities. We finally find that the share of right‐wing votes in favor of the Swiss People's Party appears to be more elastic with respect to the share of culturally different immigrants than natives' attitudes themselves, suggesting that the party has gained a disproportionate vote share from attitudinal changes caused by immigrant inflows.