Two sides of Swansea’s personality glare at each other across the bay. At one end is Mumbles and the pretty coastline, a necklace of lights along the pier, Catherine Zeta-Jones’s massive house. And on the opposite side, looking unimpressed, is Port Talbot and the steelworks: a knot of pipes and smokestacks filling the sky with yellow, blue and sometimes green flames. I grew up on the hill halfway between the two and, depending on my mood, I can see two completely different versions of my childhood.

From one angle, it was an endless summer spent swimming on Three Cliffs Bay, my sunburnt friends and I building fires, climbing rocks and trying to get off with each other. Looking the other way, I remember traipsing through a wet, grim city that could be genuinely frightening, a place that was famous for two things: car theft and football violence. I still can’t quite believe that my father – a bespectacled history lecturer who was known to wear actual elbow patches – often took me to watch the match from the North Bank of the Vetch Field stadium, to stand quivering among the furious skinheads, praying that they wouldn’t hear his English accent. At night, I would sometimes be woken by the stolen cars screeching down the steep road by our house. In the 90s, Swansea was called the car crime capital of the UK. They eventually had to install concrete bollards at the bottom of the hill to stop joyriders careering through people’s bay windows.

‘Of course, when I was growing up in Swansea, I wanted to escape’ … Joe Dunthorne. Photograph: Linda Nylind/The Guardian

Swansea thrives on its contradictions. Dylan Thomas, who grew up on the same hill, called it a “lovely, ugly town”. Many years later the film Twin Town, which is about joyriding in Swansea, updated that phrase to “pretty shitty city”. I’ve always felt that, in a sense, the shittiness protects the prettiness. The steelworks, for example, are a literal smokescreen to stop outsiders from noticing the most beautiful landscape on Earth.

Of course, when I was growing up in Swansea, I wanted to escape. I wanted to become a writer. I wanted to see the world. All my early, terrible fiction was set in exotic-sounding places I’d never been to: Santiago de Compostela, Tokyo, Oaxaca. It wasn’t until years later when I finally wrote about my hometown that my characters felt believable. It was exhilarating to discover that I had the whole of Swansea stored in my brain, like a multisensory Google Streetview. All I had to do was drop down my inner pegman and, instantly, there was life.

Though the city’s hard edges have softened a little over time, it’s a relief that it has kept its cracked personality. You can still find perfect empty beaches. The steelworks still fill the sky with fire. And if you’re lucky you might still see boys from the hill riding wild horses down into the city centre, trotting bare-chested past the big McDonald’s. Though that’s a detail you could never include in a short story. No one would believe it.

• Joe Dunthorne’s The Adulterants is published by Hamish Hamilton. His debut poetry collection, O Positive, will be published by Faber in April.