Boris Johnson was still a backbench Conservative MP and Donald Trump was a property developer and reality television star in the summer that Greetings from Bury Park was published. It was June 2007. I was 36, a journalist and broadcaster living in London. The world I worked in was white, middle-class and metropolitan – a long way from the world in which I had been raised. I had grown up in Luton, the working-class son of Pakistani parents. My father arrived in Britain in 1963 and my mother followed 11 years later with their three children. I was almost three years old. Bury Park was the Asian district of Luton, and my father worked on the production line at the Vauxhall car factory while my mother was a seamstress at home. My childhood was defined by a lack of money and a vivid awareness that my future was limited by my class and my colour. When I managed to get to university and build a career in the media, it became apparent that what I had considered an ordinary upbringing was very different from those of the people I worked among.

It was also striking that I never saw lives like mine depicted in popular culture or in books. Working-class lives, Muslim lives, lives defined by their apparent ordinariness. When I started working on a memoir, I did so with the ambition of opening up the world in which I had been raised. I pictured my mother, Rasool Bibi, walking along a street in Bury Park in her traditional shalwar kameez. What would a white person, someone who could not speak Urdu, think of her? What questions would they wish to ask her if they could? I set about writing my book with the hope that by writing very specifically I might tell a more universal story. Perhaps it was possible that sharing my family’s history would help to normalise this immigrant tale, and confirm that stories like ours belonged within the larger narrative of British history.

The suggestion that, being brown-skinned and Muslim, I would never be fully British, was reinforced by my own parents

The book was, at its heart, the story of a brown boy who desperately wanted to belong. I grew up at a time when racist football fans would run through Bury Park smashing shop windows and abusing anyone who happened to be on the street. My parents would warn me to get home before kick-off in case things kicked off. On television I would watch Tory politicians such as Norman Tebbit question the loyalty of folks who looked like me. White boys would urinate through the letterbox of my friend Amolak’s home. The suggestion that, being brown-skinned and Muslim, I would never be fully British, was reinforced by my own parents. My father would tell me that Pakistan was my true home even though I had left before I was three. He would say that white people would never accept me – there was no point in trying to integrate because I would never belong. Whenever I told myself or others that I was British, there was a nagging sense that I was a fraud. My right to say I belonged in this country felt fragile.

It was not until the election of Tony Blair in 1997 that this fragility began to be replaced with some sense of ease and confidence. I vividly remember the Observer headline on the first weekend of the Blair government, declaring “Goodbye Xenophobia”. Nine months after Blair was elected, indie band Cornershop were No 1 in the singles charts with “Brimful of Asha”, a song about a female Bollywood singer. The old certainties seemed to be giving way to exciting new possibilities. By the time Greetings from Bury Park was published, I was convinced that the arc of British history was bending towards tolerance. I was wrong.

Facebook Twitter Pinterest Luton life … from left: Kulvinder Ghir, Meera Ganatra and Viveik Kalra in Blinded by the Light. Photograph: Landmark Media/Alamy

The book was welcomed as an insight into the identity struggles for second generation British Muslims. After the 9/11 attacks there had been a spate of books – notably The Islamist by Ed Husain – that sought to understand Islamic radicalisation through personal stories. My story was also about rock music, specifically Bruce Springsteen (one of whose albums, Greetings from Asbury Park, inspired the title), which meant that it attracted readers new to books about the migrant experience. From a young age, I had sought escape in the work of American writers, television shows and in particular Springsteen’s music. His songs reflected a working-class experience that echoed mine. He sang about fathers and sons with an honesty and empathy that made me reflect on my own relationship with my father. He also articulated a generous version of American patriotism that suggested the US was an inclusive and welcoming place. In America they hardly knew what Muslims were, I told myself, and had most likely never heard of Pakistan. If life in Britain became unbearable, there was always the US.

And then came 9/11. In its aftermath my love for America waned. In the book I recount a conversation with Amolak, in which we resigned ourselves to the reality being over of the days when Americans barely knew what Muslims were. Yet, one year after Greetings from Bury Park was published, Barack Obama, then a state senator from Illinois, became the Democratic party’s nominee for president. His victory that year restored my faith. The Obama presidency was confirmation, it seemed at the time, that in the US anyone could fulfil their dreams, even someone born to a Kenyan Muslim.

The film option to my book was sold in 2012. It is hard to process how much the world has changed in the seven years since. The film, Blinded by the Light, will be released in August. Directed by Gurinder Chadha, of Bend It Like Beckham fame, and adapted by me, with Chadha and her husband Paul Mayeda Berges as co-writers, it is fictional but emotionally autobiographical. It stays loyal to the heart and soul of my book. The story is about identity and belonging, about parents and children and how we children of immigrants wish to stand on our own feet without trampling on our parents’ hearts. It is also the story of the racism of my childhood and the possibility of change. I had hoped that the tales of my teenage years would, with each year, read more and more like a dispatch from a long gone era. Younger readers would greet them with horrified fascination and a renewed gratitude for how much things had changed. It turns out that things have not changed as much as I had hoped. The old fragility has returned.

Facebook Twitter Pinterest A quest to belong … Sarfraz Manzoor. Photograph: Karen Robinson/The Observer

In the 12 years between my book being published and the film made, xenophobia has not only returned, it has become normalised. The racism that hovered ominously in the background of my teenage life, and sometimes in the foreground, has returned. In the days after a man ploughed into worshippers outside a mosque in Finsbury Park in 2017, only minutes from where I live, I had to take my daughter on a different route to her violin lesson, so she wouldn’t see the aftermath of the attack. In my teenage years it was the National Front who marched through my hometown; now it is Britain First and the English Defence League. Tommy Robinson, another son of Luton, who founded the EDL, is welcomed into Ukip and Johnson, the man who may be our next prime minister, freely uses mocking language about Muslim women who wear niqabs. The right of anyone who is not white to say they are fully British is once again being challenged.

So how will the film look to viewers today? Will the message of hope still resonate here, or in the US, where the election of Trump has overturned the happy ending of the Obama years? When it premiered at the Sundance festival, the film landed one of the biggest deals, for $15m from Warner Bros. Earlier this year I flew to Hollywood, Las Vegas and New York to attend preview screenings, and the responses of those who saw it were fascinating. After the Hollywood screening, a young woman approached me and told me she was raised in a Baptist community in Tennessee and had wanted to chase a dream of being a singer songwriter but her parents disapproved. “Your story was my story,” she told me. In New York I met a couple who had come for the Springsteen plotline, but had been affected by the story of a boy wanting to belong. They said they did not know anyone Muslim, but that I was “basically just a Pakistani version” of them.

We are living in a time when many of the battles I believed had been won are having to be refought. But the strongest weapon against those who seek to sow division is empathy. That this film is being made now, that it will reach audiences who haven’t read the book, gives me a reason to feel hopeful.

• Greetings from Bury Park by Sarfraz Manzoor is published by Bloomsbury. Blinded by the Light is released in cinemas on 9 August.