As England play their first games of Euro 2012, I'd like to be an England fan. But even though I'm English, it's hard.

For a start, as a British Muslim, I am unsettled by the sight of England supporters dressed as Christian knights and jovially waving Crusader shields at the European championships in Poland and Ukraine. Footage of last night's cagey opener with France was interspersed with close-ups of young men dressed in the armour of Knights Templar hordes. There's an irony in the fact that images of Polish supporters chanting antisemitic slogans and giving Nazi salutes have been met with such deserved outrage, but to brandish a sword and recall the brutal and bloody invasion of Muslim lands is portrayed as harmless banter.

There is an obvious difference of course. I don't for a second believe that those dressed as Christian knights do so to offend Muslims – I hope not, anyway – whereas there is obvious menace in the sickening behaviour by neo-Nazis. But this doesn't make it any less disturbing an image for the Muslims in this country and around the world.

The Crusades are romanticised in the west as heroic battles to win back the holy lands in the name of Christianity. But for Muslims they are remembered as two centuries of brutal and unprovoked attacks on Arab lands. To celebrate this in fancy dress recalls a bloody and divisive chapter in Muslim-Christian relations. That may not be the intention of those donning the fake chainmail and helmets but there's no denying it's a uniform of war that certainly doesn't instil a feeling of inclusiveness in me.

I wonder how England supporters would react to scenes at the 2022 World Cup in Qatar, were masses of Arab fans to dress up in Saladin turbans and brandish Saracen swords emblazoned with Qu'ranic verse. I'm not sure it would be seen as friendly high jinx.

As a British Asian, moreover, I find it very difficult to get behind Team England because I abhor the fact that it includes an individual facing criminal charges for racist abuse in the workplace. The fact that the accuser's brother is missing from the squad and hasn't been picked on merit makes me feel extremely uncomfortable about supporting my country. John Terry may be innocent until proven guilty, but he and his legal team forfeited an opportunity to clear his name by requesting a delay to the court case until after the tournament. If I was facing such serious charges, I would have been suspended pending trial. I find the FA's decision to strip him of the captaincy and yet allow him to play both inconsistent and disgraceful.

I've been among England supporters in the past and felt less than welcome. They've chanted various songs either in my presence or even in my direction. I particularly enjoy the pointed lyrics "You never had an empire!" – and I suppose I should be flattered that so many of my fellow countrymen would rather be of Pakistani origin than Turkish extraction.

It goes hand-in-hand with the jingoistic nationalism we're subjected to by certain sections of the press. Not for nothing is it called a campaign. On one level it's just plain xenophobia. On another level it's a handy reminder that we were at war with many of these beastly foreigners at one time or another, and we killed lots of them and came out on top. I say we, although I'm not entirely sure I want to be included in that number. And I guess the feeling's mutual.

Don't get me wrong, the John Terry issue is the obstacle for me in getting behind the England team. It genuinely offends me and it sends out a horrible message. It tells me that if I'm racially abused by someone at work who is seen as good at their job, I should keep my mouth shut. I am sure the whole Crusader-fancy-dress-thing is done in all innocence. But in the context of the English Defence League and their anti-Islam rhetoric it doesn't exactly fill me with patriotic fervour. It makes me feel Muslim, rather than English, and I'd much rather feel both.

• Follow Comment is free on Twitter @commentisfree