After seven tumultuous years reporting from Pakistan, Guardian correspondent Declan Walsh reflects on the inspiring figures, the jaw-dropping landscapes, the deep corruption – and the day the Taliban came to town

Even before you reach Pakistan there's reason to fret. "Ladies and gentlemen, we will be landing shortly, inshallah," says the Pakistan International Airlines pilot, 10 minutes outside Islamabad. To the western ear this ancient invocation – literally "God willing" – can be disconcerting: you pray the crew are relying on more than divine providence to set down safety. But these days it's about right – Pakistan, a country buffeted by mysterious if not entirely holy forces, seems to have surrendered to its fate.

Viewed from the outside, Pakistan looms as the Fukushima of fundamentalism: a volatile, treacherous place filled with frothing Islamists and double-dealing generals, leaking plutonium-grade terrorist trouble. Forget the "world's most dangerous country" moniker, by now old hat. Look to recent coverage: "Hornet's Nest" declares this week's Economist; "The Ally from Hell" proclaims the Atlantic.

Western condemnation has a moral quality, the tinge of wounded betrayal. Much of it is rooted in Afghanistan, where many blame Pakistan for the Taliban resurgence. Some years ago a senior UN official in Kabul warned me the US could launch unilateral airstrikes if Pakistan didn't get into line. Surely it would be unwise to destabilise a nuclear-armed country of 170m people, I said. "Well," he shot back grimly. "Maybe they deserve it."

Yet for all the stone-throwing, hard facts are elusive. Did the powerful Inter-Services Intelligence (ISI) spy agency really shelter Bin Laden? Does it control the notorious Haqqani network? Did it play a role in the 2008 Mumbai attacks? If smoking guns abound, the Pakistanis are remarkably good at wiping their fingerprints from the trigger. Instead, we are left with a murky stew of allegations, coincidences and the steamy whispers of western spies.

Perhaps the embodiment of this conundrum is Pervez Musharraf, the former military ruler once beloved of the west. In a recent interview the BBC's Stephen Sackur harangued him about Pakistani perfidy. What of the Taliban safe havens? Sackur demanded. Or the Quetta Shura? Or reports that the monocular Taliban leader, Mullah Omar, resides happily in Pakistani suburbia? Musharraf sat through the mauling, visibly bristling, then shot back. "You say it is true. I say it is all nonsense," he said, wearing his trademark wounded-puppy face. "This is a mirage. This is what people say. This is what you think." [My italics].

But what should we think – conspiracy, cockup or thinly veiled chaos? Puzzling out the answers to that question has been central to my seven years reporting from Pakistan for the Guardian. Much of it was dominated by the banner dramas: bombs and political heaves, spy scandals and shootings. But there were also, I discovered, truths to be gleaned from the smaller things – such as the way people drive.

Pakistanis swerve into heavy traffic without looking, don't stick to their lane or indicate, which makes it hard to predict where they are coming from or going to. Social graces are rare – horns honk, headlights are impatiently flashed – but social hierarchy is observed: hulking four-wheel drives (increasingly armour-plated) barge through the swarms of matchbox cars. Off to the side, the police are taking bribes.

But pull off the road and everything changes. Pakistanis are welcoming, generous and voluble. They insist you stay for tea, or the night. They love to gab, often with glorious indiscretion – national politics and local tattle, cricket scandals, movie stars and conspiracy theories. This is fun, and good for the business of journalism.

While Islam is technically the glue of society, you learn, the real bonds are forged around clans, tribes, personal contacts. To get anything done, the official route is often pointless – the key is sifarish, the reference of an influential friend. Journalists use sifarish a lot; occasionally they are called on to dispense it too.

Late one night, shortly after the last election, I got a surprise phone call from a ruling party official. Previously chatty and relaxed, he spoke in a loud and oddly deliberate voice. "Do you remember that place you mentioned last night – the 'Cat House'?" he said. I remembered no such thing. "Well, the police have turned up," he continued. "And I was hoping you might have a word with them." Seconds later the line dropped; I didn't call back.

Two days later the papers carried reports of a police raid on a speakeasy-cum-brothel in a smart part of Islamabad, called the Cathouse. They arrested Russian and Chinese women, dozens of bottles of liquor and a number of punters – including a newly elected ruling party MP and his entourage, including my friend. But they were released without charge, the reports noted, after a phone call from a "higher-up" in government.

I thought that was the end of the matter until a police video of the raid surfaced on the internet some months later. It showed officers storming into the Cathouse, arguing with Russian women and at, at one point, a middle-aged man in a crowded corridor, shouting into his phone.

"Do you remember that place you mentioned last night?" says my friend "The Cathouse?"

Such laughs have been regrettably rare. When I arrived in 2004, Islamabad was a somnolent, reliably dull city. By night, the sons of the rich drag-raced their daddies' cars along deserted streets, swerving to avoid wild boar ambling from the bushes. Foreigners mocked the capital for its provincial feel. "Islamabad – half the size of a New York graveyard but twice as dead" went the diplomats' tired gag as white-gloved waiters served gin and tonic on manicured lawns.

Then the Taliban came to town. It started with the bloody siege of the Red mosque complex in July 2007, just before Pakistan's 60th birthday. Bullets zipped through the leafy streets; I dusted off my flak jacket. Then came the bombs: at markets, checkposts, the Naval headquarters, UN offices, the five-star Marriott hotel. Up the street from my house, Benazir Bhutto gave speeches from behind barbed wire, during a brief-lived house arrest. Weeks later she drove out to Rawalpindi, where she was assassinated.

Today the blasts have stopped, mostly, but the city is cloistered in concrete. Fortified walls rise over the streets, vehicles slalom through elaborate checkposts, hotel entrances resemble prisons with gold-buttoned guards. Embassies are retreating into a sandbagged, Green Zone-style enclave; the presidency and even ISI headquarters are similarly isolated.

That, however, is just the cosseted capital – the real pain has been felt elsewhere. Pakistan has paid a high blood price for what my guardian colleague Jason Burke calls the "9/11 wars". Since 2001, up to 5,000 Pakistanis have died in more that 300 suicide attacks; the victims range from toddlers to three-star generals. Another 13,000 have been wounded. This is partly the legacy from the military's decades-old dabbling in Islamist extremism, but for most Pakistanis the culprit is America.

Television shows fizz with anti-American anger; many say the "Ally from Hell" epithet applies to the US, not them. Things have never been worse: outrage at the killing of 24 Pakistani soldiers in a murky border incident triggered a blockade on Nato supplies, the closure of a CIA drone base and the boycott of a conference on the future of Afghanistan – and that's just in the last week.

Washington, meanwhile, is moving to restrict $700m (£450m) in aid. The relationship is beset by frustrations and misunderstandings on both sides, but the net effect is that Pakistanis are more profoundly isolated from the outside world than they have been in decades. This cannot be good.

Women dancing at the Sehwan Sharif festival, Pakistan, one of the largest Sufi gatherings in South Asia. Photograph: Declan Walsh

Many Pakistanis – educated, ambitious, modern – resent being lumped in with the terrorists. "Why don't you write about the other Pakistan?" is a frequent refrain – "other" being the country of software companies, pizza dinners, effervescent art shows and quality literature. When I could, I did, with a tendency towards the counter-intuitive: the booming brewery across the street from military headquarters; the transvestite civil rights movement, the punk rock bands and oxygen bars and rambunctious polo tournaments in the heights of the Hindu Kush. But perhaps the most memorable experiences were rooted in the rich cultural and religious heritage. One of my best trips was in Sehwan Sharif in Sindh, a glorious Sufi festival on the banks of the Indus with a mesmeric mix of party and prayer – a spectacle to make the head spin and the heart sing.

Still, there's no getting around it: Pakistan is beset with problems that no amount of jolly beer stories or whirling dervishes can remedy. It is, as a psychologist might say, a country with serious issues. Most are decades old – the overweening army, the confused place of Islam, the covert support for jihad, deep-rooted corruption, the poisoned bond with America. Resolving them has never been so urgent.

One reason is Afghanistan. As western troops draw down by 2014, Pakistan can help construct a stable future for the war-ravaged country – or spoil a deal it dislikes. But beyond that, it is the internal stability of Pakistan that is more worrying. The country is riven by ethnic, tribal and political faultlines, which, in turn, are being exacerbated by galloping population growth and deepening poverty. Turmoil in a country with at least 120 nuclear warheads and a projected population of 300m people by 2030 could make Afghanistan look like a walk in the park. Talk of a "nuclear Somalia" is overstated, but you get the point.

Yet there is little sign of revolution. As the Arab spring swept the Middle East, Pakistan was quiet because, in a sense, it already has what others are demanding: elections. The problem is that few like the results. Asif Ali Zardari, the accidental president, suffers a crippling legitimacy deficit driven by perceptions of corruption and a more fundamental struggle for supremacy. Just a few years ago the army chief, General Ashfaq Kayani, mused to the US ambassador about the possibility of a coup. Last week, Zardari suddenly flew to Dubai, triggering fresh speculation that such an upset was about to happen. The hype seems unfounded, and Zardari says he will soon return. But few doubt Kayani is the real power.

Will ordinary Pakistanis tire of this power game? While there is no sign of a spring tide, millions of tiny waves are lapping the shores of despair. In October, Raja Khan, an unemployed man from Sindh province, travelled hundreds of miles to Islamabad. Standing outside parliament he doused himself in kerosene then struck a match. Hours later, racked with pain, the 23-year-old died. Poverty had ground him down, Khan said in a farewell note. As his coffin was nailed shut, his wife gave birth to their third son. His elderly father cried out: "Oh, Zardari, where are you?"

It's not just Pakistan – over the seven years foreign correspondence changed drastically, too. In 2004, the Guardian focused on UK readers; today, through the internet, our audience is at once global and intensely local. Pakistanis leap on every story, scrutinising and commenting, particularly on Twitter, a medium many have embraced with gusto. It helps to project less obvious stories, such as a feature on the appalling wave of alleged state-sponsored killings in Balochistan earlier this year. But the intriguing feedback I received came in the form of an old-fashioned letter. Charles Burman was 92 years old, a former British army signals sergeant who had fought a long forgotten colonial campaign in the tribal belt in the 1930s and 40s. In wobbly handwriting he sent a fascinating account of his experiences; Waziristan was pretty dangerous back then, too, it turns out.

Not everyone liked the coverage. Fatima Bhutto, niece of the assassinated Benazir, once suggested I was "on the PPP payroll", referring to the government party; pro-government blogs suggested I was peddling the ISI line; the ISI monitored my phone calls and occasionally rang to voice its own displeasure. The US military in Afghanistan blacklisted me briefly; the Taliban called with a ransom demand for a kidnapped hostage; Pervez Musharraf threatened to sue. That was all fine – multi-directional criticism is a compliment – but sometimes the story came a little too close.

In 2008 a Guardian fixer was abducted and tortured while investigating a story on intelligence agency abduction and torture. Last year, for a few nail-biting hours, a close friend's father was caught up in a brutal gun-attack on a mosque belonging to the minority Ahmadi community in Lahore. He survived but more than 100 others died. The bombings took a toll. A few minutes after the 2008 suicide bombing of the Marriott, a hotel where I got my hair cut and had coffee with contacts, I found myself standing in the rubble, dazed by the enormity of the atrocity. A giant crater occupied the park, staff in bloodstained uniforms stumbled through the lobby, hunting for survivors, orange flames licked the ash-laden sky. Blood squelched underfoot.

Retreating outside I found a preppy looking young man sitting on the verge, staring numbly into the inferno. His name was Ehsan Peerzada and he was 19 years old, articulate and educated, the son of a senior civil servant. In other circumstances, I might have interviewed him for a story on savvy, westernised Pakistanis. Now he railed in a stream of invective against everyone – Islamist extremists, Americans, drone strikes – struggling to make sense of it all. "It's not fair," he mumbled. "It's not fair."

Pakistani hijras, or transgender men, at a function on the outskirts of Rawalpindi, the garrison city that is home to the headquarters of Pakistan's powerful military. Photograph: Declan Walsh

It's not all darkness; away from the bang-bang, life in Pakistan can be richly rewarding. I've been humbled by inspiring figures, traversed jaw-dropping landscapes and attended some wild parties, on one occasion with a roomful of transvestites. Where else can you find yourself with a bearded, joint-rolling characters, as I once did in Peshawar, nicknamed "Mullah Omar"? Even the news can be fun. Some years ago the cricket board issued a press release detailing "genital warts" of its errant star, Shoaib Akhtar. These days, bomb stories vie for space with Veena Malik, a daring actress who appeared topless wearing nothing but a tattoo that read "ISI". Malik has denounced the pictures, claiming – but of course – that they are the product of conspiracy.

I hoped that my reporting portrayed the rich complexity of a society that, below the surface, defies its stereotypes. But on some occasions there was just nothing to be said. A few months ago I visited a house in Rawalpindi with a giant poster over the windows, depicting a heroic warrior on a gallant white steed. The warrior was Mumtaz Qadri, the police bodyguard who gunned down the Punjab governor Salmaan Taseer, last January, and this was his house.

Outside, young children shouted slogans for Qadri, a curly-bearded extremist who killed Taseer because he championed the case of a poor Christian woman who had been prosecuted under the country's notorious blasphemy laws. Others joined them, protesting against Qadri's prosecution for murder. The air was thick with talk of persecution. "Qadri is a great martyr," said one man. "What he did is permitted by Islam." Then the crowd poured through the streets and on to the highway leading to Islamabad. The police closed the road and watched.

The celebration of Qadri, a deluded fanatic, was deeply depressing. So was the fact that nobody dared raise their voice against his supporters, not even the government. Instead, the judge who sentenced Qadri has fled Pakistan. Aasia Bibi, the Christian at the heart of the furore, remains in jail. And Taseer's son, Shahbaz, has been kidnapped – probably by Qadri sympathisers. An ugly spectacle, it betrays questions about something deeply unhealthy at the core of Pakistani society.

Still, many Pakistanis have similar doubts. There is a striking amount of national introspection in a hearteningly vibrant press. But which way out of the quagmire? Imran Khan, the cricket star turned political sensation, says he has answers. He exudes the confidence of a man who believes his time has come.But his ideas are controversial and, critics say, naive. His stance against "politics" echoes that of Musharraf a decade ago – a perception he will have to work hard to dispel.

People often ask the most basic question about Pakistan: will it survive? The question has been going round for decades; the naysayers inevitably silenced. Is the current situation any more precarious? The country has deep stores of resilience, but is more vulnerable to external shocks than ever before. One thing, however, is clear: inshallah may have worked until now, but it is no longer enough.

Declan Walsh's book Insh'Allah Nation is out next year. twitter.com/declanwalsh