So there I was walking home from work yesterday and I found myself in the middle of anarchist riots in the capital. I was wearing a suit and a poppy so I was apparently deemed a natural target. Worse, I was a non-Caucasian wearing a poppy.

“Why are you wearing that, Tory boy?” some of them shouted at me. There must be something about me that seems to attract anarchists. When a group of them saw me entering the 2013 Tory conference in Manchester, they shrieked that I couldn’t be a real member of the party as I was “brown” and looked like “Uncle Tom”. I remember to this day the hatred that contorted their faces.

The press made much of the incident, saying it was an outrageous and unacceptable act of racism, but, if there is one thing I’ve learnt about being a modern Tory, it is that you need to develop a thick skin.

It was apparent to me that the best way to get people out of poverty is the free market system

To be open about your political affiliations – as I’ve always been through years of canvassing and fundraising – can also involve physical assaults as well as verbal ones, especially when you don’t, like me, actually look the part. One incident occurred outside King’s Cross Station, which followed my party’s election victory earlier this year. A group of young men cornered me, pushed me to the floor and kicked me around a bit.

There are those who wonder why I keep putting myself through it. It was Winston Churchill who said, after all, that you can gauge the righteousness of your cause by the ugliness of the abuse it provokes.

I only wish I could sit down and talk to these people who have shrieked at me and kicked me and talk through with them how I have come to hold the views that I do. Certainly I’d accept that my politics seemed from the very beginning to be a matter of waywardness, if not downright eccentricity, since I am the son of an Iranian refugee who was brought up on a council estate in Lewisham, south London.

There it was taken as read that the socialists were the good guys and the Tories the baddies. There was a nihilism and a sense of entitlement on that estate that I was quick – even as a teenager – to see through.

The country that had given my family refuge had given us everything that we possessed – including a sense of hope about the future – just as my old country, in the grip of fundamentalists, had taken it all from us.

Maybe it is because I had experience of what happens in top-down countries – where the little man never dares to say a word against the all-powerful state – that the Conservative party appealed to me. It so happens it was a Labour man, David Blunkett, who said that if you find yourself at the bottom of the social scale – living in poverty – the last thing you want to do is to wallow in it: you want to get out of it as soon as possible.

It was apparent to me that the best way to get people out of poverty is the free market system. It would have been my mother’s dream, as a young woman growing up in Iran, to have a government that cut taxes, that provided jobs and simply allowed people to get on with their lives. She never expected government handouts, but merely an opportunity to experience prosperity through individual liberty.

‘That’s what I still love about Conservative politics: I see it as a platform where, whatever your background, it’s your ideas that count.’ Photograph: Toby Melville/AFP/Getty Images

As a classical liberal Tory, I want people to do what they want, as long as they are not harming others. It is not the government’s job to tell you who you should marry, it is not the government’s job to limit your chances to start your own business, and it is certainly not the government’s job to try to influence behaviour through social policies. A great example is a possible tax on junk food, an attempt to discourage unhealthy diets among young people. It is not the job of the government to bring up our children. It is the job of parents to be parents.

I love the fact the British are by nature resentful of authority. This is the culture that made this country one of the first few to introduce the concept of “government by consent” during the Anglo-Saxon period. The system has progressed and evolved to give us what we have today: a democratic constitutional monarchy.

People need to get that Tories don’t look like ruddy-faced old white men in exclusive clubs. They look like me. I recall my first experience of a political meeting at university in Manchester. A friend who was an activist in the Labour group there took me to one of their meetings.

I spent the whole night listening to the students talk about my own heritage and how the Tories are worse than the Iranian regime and that the Labour party will protect me. It was precisely because they were so patronising towards me – and knew so little about what they were talking about – that a few days later I proudly joined the Conservative society. There my ethnicity was never mentioned. They respected my right to wear a poppy as they respected my right not to.

They simply discussed ideas and that weekend’s football fixtures. I was treated as an individual and someone who had a life outside of politics. What I had to say – even though it did not always chime with others’ views in the party, which is a big tent – seemed to matter. That’s what I still love about Conservative politics: I see it as a platform where, whatever your background, it’s your ideas that count. The openness and common sense that characterises Conservative debate hooked me – and will continue to hook others like me, so long as Labour continues to offer the opposite.