Everybody has their favorite story about those wacky North Koreans. One popular tale passed among Pyongyang's small band of intrepid diplomats is about the short-lived Australian Embassy, which was closed a few years ago soon after it opened. It was said that the North didn't appreciate Australia's offer to mediate with the South.

In fact, the story goes, that wasn't the reason at all. The fledgling embassy had thrown an opening party, a toga party. There aren't many cars on Pyongyang streets, and the sight of the tiny foreign community criss-crossing town wrapped in white sheets was too much for the grim-faced Stalinists. "They weren't sure what it meant, but were certain of one thing - it was a conspiracy," one diplomat told me.

I was attending a diplomatic party, one of four parties in the city that night, which the host thought was a record, there being only 200-odd westerners in North Korea, a figure further reduced by the government's decision to throw out aid workers at the end of last year.

Word at the Random Access Club, the bar in the World Food Program building (so-called because of the distinct lack of random access enjoyed by the WFP to inspect their programs), was that the "Dear Leader" found even this number of westerners too intrusive for his liking, and if their departure meant more hardship for a country that had already seen hundreds of thousands of deaths from famine, then so be it.

It's a pity, said another diplomat, who'd been improving his handicap on Pyongyang's new golf course (designed, the North claims, by Dear Leader Kim Jong Il himself, who apparently got 11 hole-in-ones on his first outing). He feared he'd now struggle for partners, "and there's little else to do at the weekends."

That visit, in September 2005, had been my third, and was ostensibly to cover the Mass Games, the biggest propaganda exercise on the planet, involving more than 100,000 participants, including a human "wall" of children flipping cards to create giant pictures down a whole side of the stadium. The games lasted a month, and marked the 60th anniversary of the ruling communist Korean Workers party. Each night the 150,000 seat May Day stadium was packed with workers and farmers bussed in for the two-hour show, telling the story of a strong, modern and prosperous country, a country of abundant harvests, happy farmers and plenty of missiles.

That picture (with the possible exception of the missiles), of course, couldn't be further from the truth. The reality is an impoverished and brutal dictatorship, and even as the Mass Games were being organized, the government had given the notice to aid agencies to leave the country by the end of the year.

At the same time the authorities had curtailed a limited mobile phone network, which had anyway only been available to top officials and selected diplomats, and they'd closed down farmers markets that a year earlier – during another of my visits - had been hailed as the first signs of possible China-style reforms.

Diplomats were trying to figure out what it all meant. Was the Dear Leader losing his grip? Was the army calling the shots? Every diplomat seemed to have a different take on what it all meant. The Russians claimed to have an inside track – Kim was being pushed aside, they said, but others dismissed that as wishful thinking, since the Russians no longer had any real leverage.

One thing they did all agree on was that China held most cards – supplying the majority of the country's fuel and food supplies, but they doubted Beijing would rock the boat, not wanting to destabilize the regime. That was, of course, before this week's apparent nuclear test, and the enormous loss of face this entailed for Chinese diplomacy. And even then, diplomats were asking whether Kim would even listen to Beijing.

It is a measure of just how hard it is to get under the skin of this secretive and paranoid country, that even the most seasoned of diplomats were struggling to make sense of what was going on, and falling back on their rich armoury of stories about the mad, bad Dear Leader.

But is Kim really mad? Sure, he looks the part, but most of what he does is not irrational, given the limited number of cards he has and the paranoia, cultivated by events like the mass games, and in endless television propaganda, films and billboards about "evil" America and the brutality of the Korean War. Hardly a day goes by without the Korean people being reminded about the war and warned, not just that they are under threat, but that a nuclear attack could come at any time.

And in such a closed society – there is no public access to the Internet or media from outside the country - there is hardly any alternative sources of information for a struggling people, told that every hardship is the result of outside powers.

There is logic behind Pyongyang's actions. Far from being simply mad or bad, they look at what has happened in Iraq and believe their very survival is at stake. That doesn't make them any cuddlier. It's a nasty regime, but it is a better way of understanding them.

Part of the problem for a journalist (as well as those diplomats) is simply getting access, getting to talk to ordinary North Koreans. The regime is so secretive, that it feeds all sorts of speculation – most of it well wide of the mark.

During my three visits I've tried to get a sense of how "real" North Koreans see the world. It is a tough exercise. I was always assigned two foreign ministry minders, who traveled everywhere with me. On the rare occasions I sneaked out of my hotel, I was followed, suddenly bumping into one of my minders in the shop were he too happened to be browsing.

Pyongyang is a grim, grey city of faceless apartment blocks, wide boulevards - and hardly any traffic, though plenty of pirouetting traffic policemen. Its also a good deal better off than elsewhere in the country. Everywhere there are large billboards praising the Dear Leader and his father, Kim IL Sung, the founder of the country, known as the Great Leader. He's been dead for more than a decade, but remains head of state.

I found plenty of curiosity from North Koreans I spoke to, but any conversations are closely monitored by the minders, though even without them, I'm not sure you'd find any critics. This is such a closed society, largely cut off from any contact from the outside world, its people bombarded with the most poisonous propaganda, especially against the United States, which is depicted in posters, and in the media as a blood-thirsty bully, intent on going to war with the North.

It's my belief that the Dear Leader doesn't just see nuclear weapons as a bargaining chip to trade away for aid and security guarantees, but that he really want them. Having looked at what happened to Iraq, he seemingly decided that these weapons are the ultimate guarantee of their survival. This makes them a far more dangerous adversary.