Tommy was much shorter than I anticipated. Surrounded by an entourage, huddled in one corner of the green room, he kept shuffling uncomfortably and throwing his shoulders back. He looked like someone who was getting ready for a fight. We didn't make eye contact, but it wasn't long before we were called down to the makeshift studio in the grammar school in Bury. It was April 2012, and it was my first face-to-face meeting with Tommy Robinson (real name Stephen Yaxley-Lennon), the leader of the English Defence League (EDL). We were appearing on a BBC1 programme called The Big Questions. Little did I know this would be the start of an 18-month journey together that would end with Tommy leaving the EDL.

It's odd the little things I remember about that encounter. It felt almost like an out-of-body experience to sit opposite the leader of the English far right in a television debate. As discussions got under way – I called for the EDL to be proscribed and talked about links between EDL ideology and the Norwegian killer Anders Breivik - I found myself distracted by Tommy's hair. It was slicked to one side as if prepared for a school photo.

Then, as the debate continued, the protests outside the studio seemed to fade and something slightly odd happened. We shared a moment of levity, perhaps humanity. As the presenter, Nicky Campbell, spoke into the camera, Tommy and I looked at each other and cracked up slightly. I'm not sure why, but Twitter noticed. Robinson was no friend of mine, and albeit condemned by people around the world as an enemy to Islam, but suddenly we were kids at the back of the class.

It was an odd position to find myself in. I had spent years as an outspoken advocate against Islamophobia, working to counter extremism and trying to address what I felt was an emerging civil rights crisis for Muslims in Britain. Muslim communities everywhere were under threat, attacks against mosques and individuals were at epidemic levels and rising. Yet the Islamic tradition is that you do not try to crush those who wish to oppress you, you try to educate them. You pray for them. You enlighten them. Despite the heated exchanges that day, I was able to extend to Tommy an offer: that we have dinner.

There was immediate uproar. Twitter erupted into a cauldron of hate as I started receiving around 200 hate-filled tweets a day - from hardcore Salafists, far left anti-fascists and, of course, the EDL. But we pressed ahead, and agreed we should film our shared journey.

I met Tommy later that month at a hotel somewhere near St Albans. He was flanked by a cameraman and was clearly revelling in his cult celebrity status - the working-class voice for the common man, little Englander, defender of the English. Tommy seemed happiest when he slotted into his groove – a well-rehearsed hustings tirade conflating Islam with terrorism, paedophilia and sharia. It hadn't been too hard to figure out that verbally attacking him in return would merely cause the barriers to go up. Islam advocates a soft-hearted, patient approach. So despite the occasional urge to let rip, I had no choice but to talk to the man, not the caricature.

Three hours of debate followed. Tommy meanwhile seemed to enjoy ordering the most expensive thing on the menu. He liked his steak on the rare side. At the end of it we both tweeted two statements from Tommy – that I "must be reading a different Qur'an to everyone else" and "if every Muslim was like you there would be no problem". The response was shocked and sceptical. That I had passed the Tommy Robinson test for acceptability was nothing to be pleased about. He had to meet more people. We needed to do more work.

In the midst of all this came the brutal killing of drummer Lee Rigby. The morning after, I joined the media junket in Woolwich. I felt we needed to not only condemn the attack but to promote unity among communities, to call for tolerance and to ask for people to search out peace. A few days later I met Tommy again, in the back streets of Newcastle, where an EDL rally was being held.

We got into a heated debate almost immediately. I challenged him, claiming he was going to stoke fear and increase tensions. He was pumped. He wasn't listening. A sizeable crowd gathered quickly at the end of the road and our debate ended up being little more than a platform for Tommy to vent. And then it happened again. We both looked at each other, sighed, laughed and before I knew it, he had put his arm around me. A little taken aback, the anti-fascist in me despised it, but the person of faith accepted it. I had to laugh. How can we begin to build a bridge if I cannot accept a gesture, however cynical?

It was only when I arrived in Bristol the next morning that Nicky Campbell showed me we had been "papped" and the picture was in the Sunday Mirror. The furore about the image spread around the world, particularly the Muslim world. It was the first and only time that I doubted the process and my conviction that dialogue was the way forward. But in the end, there was an overwhelming consensus that to accept Tommy's gesture (however cynical) was in line with the prophetic traditions, and the correct Islamic adab (manners). Voices who wanted unity and a new hope for the future were drowning out those who rejected a peace process. For the record, no – it was not a hug from me, not really.

So our journey together continued. Despite both my mother and wife questioning my sanity, I had always wanted to stand up and address an EDL meeting, and come face to face with Tommy's supporters. A town hall-style meeting was arranged at a hotel in Luton. Because of the risks, the crowd was limited to around 50 people, and I was given a four-strong security team, including my own bodyguard, a Jehovah's Witness called Rudi. It was a stressful experience. The anger and hostility from EDL members surfaced over things I thought long gone, with the National Front-daubed brick walls of 1970s Britain: coming over here and taking our jobs and our women, erosion of culture (they even believed they were limited from practising Christmas), multiculturalism, immigration. It was important to listen – they are not uncommon views. Painful ones.

At the end of the meeting, I had to break my fast, as required in the month of Ramadan. I invited Tommy back to my room and he stood with me as I offered a dua supplication/prayer. We ate food from a local Indian takeaway. Tommy's insistence on refusing halal meat on camera was a regular theme throughout our time together, despite the fact he eats it at Nandos and his favourite Turkish kebab shop. As I prayed maghrib (sunset prayers) he watched, quietly. Tommy has always been much better to talk to in a one-to-one setting. We could have a real conversation. When the camera was rolling, I felt we rarely saw the real Tommy.

One of those days when we did see him was when visiting the Aisha mosque in Walsall. In the wake of an attack on the mosque, the committee had – remarkably – invited Tommy and I to film there. We arrived to find they had prepared an enormous spread of food (while Tommy was respectful he still refused to eat) and we toured the mosque with sheikh Ibrahim Mogra, assistant general secretary of the Muslim Council of Britain, discussing the place for an enlightened Islam and mosques in contemporary society. It felt as if we were making progress.

Our last scheduled day of filming took place at a club in Mayfair earlier this month. Afterwards, I had a private discussion with Tommy. He had certainly softened on some of his views of Islam, he was better educated, but it was a question of whether he could leave the politics of prejudice behind and face the public and his tribe. He believed that the EDL would "fall apart" without him at the helm.

A week later Tommy held a conference with Maajid Nawaz, of the counter-extremism thinktank Quilliam, and announced he was quitting the EDL. I was cautiously optimistic. Throughout the journey my aim had been simple – to see if we could move Tommy on his views and to see if the British public would shift on theirs. My view had always been that any new future should be conditional on Tommy distancing himself from former extremist pals, and that shared ideology.

In the coverage that followed Tommy's announcement, I began to wonder how far he had really renounced his previously held views and whether the film had been a cynical ploy on his part. He seemed to have refined his rhetoric but little else. And there remains legitimate concern now about the mainstreaming of Tommy's far-right, extremist views.

My journey with Tommy has shown one thing – that to embrace diversity in modern society we need to work out our differences. It's often a messy and imperfect process, but it's vital that we remain hopeful. Discourse and dialogue can work. How else can we tackle hate and prejudice? The answer to hate is not more hate.

It remains to be seen how genuine Tommy's move from the far right is. For my part, I will continue to work to help reduce prejudice against Muslims and Islam in this country. But despite my reservations about Tommy, I would do it all again.

Quitting the English Defence League: When Tommy Met Mo can be seen on BBC 1 Monday 28 October at 10.35pm

• Comments for this article will be opened at 9.30am on Saturday 19 October.