Weaving up a giddying mountain track on horseback towards one of the world’s highest peaks last month, I could only reflect on how much this year’s family holiday was every bit as different as we had hoped.

With a frothing river thundering at the bottom of a vertiginous gorge on one side and alpine forest on the other, we were a long way from our usual fare of a kids’ club belting out ‘Baby Shark’.

For several years after the arrival of our daughter, holidays had become stuck in a comfortable rut. We chose a resort in the Med or the Far East, complete with disco and water slides, and spent 10 days pleasantly lazing.

Yet the memory of more varied pre-baby holidays kept nagging and a new posting to Islamabad to cover the region for The Telegraph this summer gave the opportunity for something different.

Pakistan may not yet spring to mind as an obvious family holiday destination, with a reputation that is more Homeland than The Travel Show. Yet many in the country argue, with some justification, that such a portrayal is unfair. The militant violence which blighted the country earlier in the decade has fallen dramatically following military security campaigns, and the country is raring to regain its standing as a jewel in the world of exotic and adventurous travel. The wider world is staring to take notice. British Airways last month resumed direct flights after a hiatus of 11 years. Planes currently leave three days a week, but that is expected to become more frequent. The Duke and Duchess of Cambridge are due to visit in the autumn.

As a foreign correspondent I often have to write about the grimmer things going on in a country, but like anywhere there is more to Pakistan than doom and gloom. Since arriving, we have had wonderful family weekends at the seaside in Karachi, exploring the bazaar in Peshawar, or sampling the art galleries and old city of Lahore. We have not felt threatened anywhere and have only found hospitality and curiosity. The procession of people who want to greet our daughter, pinch her cheeks and take a selfie can at times be almost overwhelming.

The seaside in Karachi Credit: GETTY

For our summer holiday, as the temperatures nudged 110F (43C) in Islamabad, we had decided to escape the heat to what is for many, the country’s finest attraction: the spectacular northern highlands.

The confluence of the Himalayan, Karakoram and Hindu Kush ranges bless the region with some of the most stunning mountainscapes and trekking in the world. Throw in the local passion for fast and furious polo, dramatic hill forts and diverse local cultures and you have quite a package.

“We have everything, but the problem is that we have been titled as a terrorist country,” said Irfanullah Baig of Waljis Travel, which arranged our trip. One of the oldest travel agents catering to overseas tourists, he makes arrangements for nearly 2,000 foreigners a year, mainly from the UK.

But beyond security, was Pakistan right for a two-week family holiday, we wondered? Was there enough to satisfy both the adults and an easily bored five-year-old?

Mountain views in Chitral Credit: GETTY

Our bespoke itinerary included 14 nights away with a driver and Toyota Hilux to handle the sometimes rough mountain roads.

Heading north from the capital, our first stop was the sublime mountain valley of Chitral. This former princely state within touching distance of the border with Afghanistan is known for its tumultuous polo, more akin to Afghan buzkashi than the formal game played elsewhere. It is also home to some enchanting hotels, namely the Ayun Fort Inn and the Hindu Kush Heights, with rooms that soak up the serene valley views. We wished we had stayed longer.

Northwards and the road becomes rougher before crossing the 12,100ft Shandur Pass, which hosts an annual three-day polo festival. Here on a plateau surrounded by an amphitheatre of mountains, teams from Chitral and neighbouring Gilgit-Baltistan battle for supremacy on what feels like the roof of the world. For the rest of the summer it is left to yaks and shepherds.

Polo at the Shandur Pass Credit: alamy

Down into Gilgit and on to Karimabad, the capital of Hunza. It is difficult to envisage a more dramatically appointed town, with views of 25,551ft Rakaposhi. All this is presided over by the wooden maze of cubby holes and history which is the 700-year-old former home of the rulers of Hunza, Baltit Fort.

On via a trip to the Chinese border in the snowy Khunjerab Pass, stopping on the way to watch scurrying marmots and majestic ibex, and then into Skardu, where each corner bludgeons the senses with a new view of elemental mountain ranges.

Baltit Fort Credit: GETTY

We ended at Fairy Meadows, reached by a nervy journey along a jeep track hugging the mountain side and then onto horses to our chalet in the shadow of Nanga Parbat, the ninth highest peak on earth.

While foreign tourists were rarely glimpsed, middle-class Pakistani holidaymakers keen to escape the scorching plains of Punjab and Sindh and see another side of their country were commonplace, many with children in tow.

Skardu Credit: GETTY

It turned out we had no reason to fear the verdict of a five-year-old. She loved it. Animal spotting, horse riding, exploring forts, and crossing rope bridges were all big hits. Long car journeys between towns may put some parents off, but for those who would rather fly, there are airports in Gilgit, Chitral and Skardu.

The Foreign Office travel advice for some parts of the country is still foreboding. Mr Baig believes it is needlessly cautious now security has improved so rapidly. “It is not realistic I have to say,” he said. “They need to revisit their policies.” But even abiding by the advice, the danger spots can be skirted around.

Pakistan has its problems with poverty, corruption and political turbulence. But it also has another side, which it is keen to show to tourists and which will repay the adventurous, whatever their age.