Coming home from a holiday last week, I had a brief conversation with the taxi driver about Belgium. He looked at me in his rear-view mirror and said, “Been somewhere lovely, have you?”

“Why yes,” I said. “I’ve been to Brussels.”

There were a few seconds of doubt and confusion. Then the taxi driver said, “Why?”

“I lived there once and I really love it,” I said.

“Oh,” said the taxi driver. And then he nodded, lost interest and put the radio on.

Belgian beauty: Audrey Hepburn, born in Brussels. Photograph: Hulton Archive/Getty Images

A plaque highlights Audrey Hepburn's house. There is usually a group of excited girls hanging around outside

Conversations about Belgium are – in my experience – often this brief, and any mention of Brussels is likely to land me in an unwanted conversation about office blocks, Brexit or a tiny statue of a naked weeing boy. But, occasionally, I’ll see someone’s eyes light up and they’ll say, “Oh, I love Brussels” – and then they’ll tell me about the friend or relative they frequently visit, or about that time, ages ago, when they lived there themselves. Brussels is that kind of place. If you actually know it, you probably love it.

My love for Brussels began in 1995. I was in my early 20s when I arrived, grubby and half-asleep, at the biggest railway station, Gare du Midi, on the night train from Avignon, having escaped a joyless job on a hotel boat in the south of France.

I’d only intended to stay for a few days on my friend Tracey’s sofa before making my way on to Zeebrugge to catch a ferry home to Felixstowe. But somehow I ended up staying a year. And even though I took an instant liking to those things for which Brussels is best known – the Grand Place, the Atomium and, of course, beer and chocolate – it is an altogether different Brussels that I fell in love with. And that Brussels is one where few tourists rarely ever venture. It’s a shame; they’re missing out.

Art deco city: Place Eugène Flagey. Photograph: Joanna van Mulder/The Observer

Ixelles is an area in the Upper Town, just south of the city centre and accessed by the metro stations Louise and Porte de Namur. I lived near the latter in the heart of the Congolese quarter known as Matongé. Having moved on from Tracey’s sofa, I was now sharing an apartment with Sandrine, a young Belgian woman. In one direction, we were two minutes away from African shopping arcades and in the other, a stone’s throw from L’Ultime Atome, a very hip cafe beloved by the British expat crowd and Belgians alike.

Mention Brussels and risk being drawn into unwanted conversations about Brexit or a tiny statue of a weeing boy

Both the arcades and the cafe are still going strong. Take a walk up Chaussée de Wavre for a little taste of Kinshasa in Belgium and then head on to Place Saint Boniface for one of the coolest cafes in Brussels. It’s worth the price of a coffee alone just to sit outside in the gorgeous little square and watch the world go by – a world which is Belgian, African and also discreetly British.

British expats have always been a presence in Ixelles. In 1842, Charlotte and Emily Brontë – on a year out, like me – regularly spent their Sundays at a house on Chaussée d’Ixelles, having tea with a British family who lived there. Later, in 1907, an English nurse named Edith Cavell arrived in Ixelles to work at a nursing school. The First World War proved fatal for Edith Cavell but she is still remembered in Brussels. There is now a street and a hospital named after her – although not in Ixelles where they perhaps might best belong.

Little Kinshasha: an African grocer’s in Matongé. Photograph: Joanna van Mulder/The Observer

Another notable British resident of Ixelles was Audrey Hepburn, who was born and grew up at 48 Rue Keyenveld, just a short walk away from one of Brussels’s most expensive boulevards, the Toison d’Or. A commemorative plaque highlights her house. It’s impossible to miss because there is usually a group of excited girls hanging around outside.

Another gem hidden down a quiet residential street is the Musée d’Ixelles. Among the permanent exhibits on display in this art gallery are paintings by Magritte, Miró and Picasso, as well as an impressive collection of original posters by Henri de Toulouse-Lautrec.

And the best thing about this museum is that you have it all pretty much to yourself. You can stand in front of a Toulouse-Lautrec for as long as you like and be confident that nobody is about to barge you out of the way and then whack you in the face with a selfie-stick. The first Belgium retrospective of the work of French photographer Robert Doisneau has just opened at the museum, and the show is running till next February

Park life: the feathery denizens of Ixelles ponds. Photograph: Joanna van Mulder/The Observer

But the art in Ixelles isn’t just in this gallery; it really is everywhere. Look up at the windows and balconies and you’re likely to spot both art nouveau and art deco features. In Place Eugène Flagey, an enormous and striking art deco building dominates the square. Having stood derelict during the 1990s, it is now an arts venue and home to Café Belga – another candidate for the coolest cafe in the world.

Just beyond it, Ixelles turns leafy and art nouveau mansions form a border along the Etangs d’Ixelles – two large and pretty ponds. It’s ever so slightly like Central Park’s Upper West Side but without any hipsters. Keep walking and you come to what may be the best-kept secret of them all, the Abbaye de la Cambre.

On a bright day, this ancient abbey and its immaculate gardens could well be the most tranquil place on earth. You have to see it to really understand. And I suppose that goes for the whole of Ixelles. But if you do go, you might want to keep it to yourself or else it’ll be as busy up there as it is in front of that dodgy statue of the weeing boy.

Way to go

Double rooms at the four-star Hotel NH Brussels Stéphanie are available from £57.35 per night (room-only) at weekends; prices can double during the week. The nearby Pantone boutique hotel has double rooms from £66 weekends, more during the week. British Airways flies Heathrow to Brussels from £70. Eurostar fares from London to Brussels start at £29 one way, if booked well in advance