If there was ever a real-life policeman who came close in progressive Swedish affections to Kurt Wallander, the bestselling creation of Henning Mankell, it would probably be Göran Lindberg, chief of police of Uppsala, the city north of Stockholm that is home to Sweden's most prestigious university. Although he lacked Wallander's humility and reticence, Lindberg was concerned, like Wallander, with the marginalised and neglected in Swedish society. He was the sponsor of a sanctuary for abused juveniles, for example, and was at the forefront of the campaign to institute a more sympathetic response to rape victims.

In particular Lindberg was a staunch enemy of sexism in the police force. He argued with colleagues, made speeches and built up a reputation as a tireless proponent of women's rights. So vocal was Lindberg that he ruffled the epaulettes of fellow policemen. "His colleagues," says PJ Anders Linder, political editor-in-chief of the newspaper Svenska Dagbladet, "were obviously not quite as obsessed with the issue as he was. He seemed to be like a civil servant who had decided that this was how he was going to make his mark."

And he did. From early in his career, Lindberg was seen by the authorities as a policing role model and was duly made the national spokesperson on sex equality in the police force. Pretty soon he established a reputation as Sweden's leading progressive policeman. So renowned was Lindberg for his political correctness and sensitivity towards women's issues that he was nicknamed "Captain Skirt". In spite of the jokes, he was rapidly promoted, becoming the dean of the police training college and eventually the police chief of Uppsala.

In January this year, following a six-month investigation, Lindberg was arrested. At the time of his apprehension he was allegedly on his way to meet a 14-year-old girl in a hotel encounter that was also due to feature a number of other men. It was said that in his car was a bag containing leather whips, handcuffs and a blindfold.

What had originally alerted the police to Lindberg's predilections was an incident in July last year in which a multimillionaire 60-year-old man was found dead beneath a balcony in a salubrious Stockholm suburb. According to police, the man had been running an illicit sex network delivering women to groups of men. Apparently on the day of his death he had been expecting the arrival at his home of an 18-year-old girl. Instead a gang of men turned up and issued a vicious beating. Shortly afterwards the man either jumped, fell or was pushed from the balcony. On the dead man's desk, investigating police found the phone number of the police chief, Lindberg.

It all reads like a plotline from Stieg Larsson's Millennium trilogy or a Wallander novel, with the striking exception that in this case it was a Wallander-style policeman who was the architect and not the detective of the crime. "The villains in Mankell's stories are all of a piece," says Lars Linder, chief cultural critic on the daily paper Dagens Nyheter. "They are scoundrels and usually connected to very wealthy or fascist networks. Whereas the thing about Lindberg is that he's so absolutely politically correct on the outside and kinky on the inside."

Last week Lindberg was jailed for six and a half years on charges of rape, pimping and procuring. He accepted that he bought sex, which is illegal in Sweden, but had denied the other charges. After Lindberg's arrest, a woman, calling herself Linda, was quoted in Swedish newspapers. She claimed to have been sexually abused by several men. "The police chief called me 'Daddy's girl'," she said. "I was told that he was important and that he would frame me if I told anyone." Again, she sounds as if she emerged, fully formed, from the pages of Mankell's fiction.

Lindberg was found guilty of aggravated rape, rape, assault, 28 counts of purchasing sex, and one of being an accessory to procurement. He was cleared of the attempted rape of a minor. As well as jailing him, the Södertörn District Court ordered Lindberg to pay 300,000 kronor (about £26,000) in compensation to three victims.

The news of Lindberg's secret life rocked Sweden. While a certain scepticism about the police is common enough in intellectual circles, the notion that the foremost advocate of women's rights in the police was in reality a serial user, and abuser, of prostitutes was enough to stun even the most grizzled cynic.

Lindberg's colleagues, and particularly his female supporters, were dumbfounded. Beatrice Ask, the justice minister, spoke of the "devastating and distressing" effect of the news. While Cecilia Malmström, who is Sweden's EU commissioner and was a member of Uppsala police board when Lindberg was police chief, said: "I have no words. I am extremely shocked. This is a man who has dedicated his career to fight for women's rights. I feel physically sick when I think about this."

In late July Stockholm was a postcard of relaxed health and vigorous prosperity. Along the spotless avenues and in the city's many green spaces, the kind of people who look as if they have escaped from a yoghurt advert took the opportunity to laze in the sunshine. The southern archipelago lightly baked under cloudless skies. Surrounded by inlets of deep blue water, the Swedish capital seemed to sparkle with a crystalline sense of benevolent purpose.

Here is the image of Sweden with which we've grown familiar, an image of which the Swedes themselves are understandably proud. It's the utopian vision of the Folkhemmet or "people's home" that, in one way or another, the Swedes have been conscientiously cultivating and exporting for almost a century.

But in recent years a darker, more disturbing picture of a failed utopia has also made its way around the world. In the 1980s Sweden began to pull back from the enormous state intervention and social reform that had guided the country for the previous half-century. And early in that transformation, on 28 February 1986, the prime minister, Olof Palme was shot and killed in the street by an assassin who has never been found.

Ever since that period, talk of a sinister underbelly, the nasty truth lurking beneath Sweden's shiny surface, has afflicted the national conversation, particularly in the cultural realm. In the novels of writers such as Mankell and Larsson, as well as the films of Lukas Moodysson, corruption, vice and despair run rampant.

All three artists (Larsson died in 2004) are avowedly leftwing and in their different ways they tell the tale of a dream betrayed, and an outcome in which the most vulnerable citizens are abandoned to a ruthless system. It's also notable that all three employ the archetype of the abused prostitute as the prime symbol of capitalist exploitation.

Moodysson's Lilya 4-Ever, made in 2002, was an unremittingly bleak account, based on a true story, of a 16-year-old girl from a former Soviet republic who is tricked into travelling to Sweden, where she is raped, held against her will and prostituted, before she commits suicide. Similar helpless victims appear in the fiction of Mankell and Larsson, where they are explicitly shown to be at the mercy of hidden, well connected and malevolent forces within Swedish society itself.

Of course, Mankell and Larsson are thriller writers, with the necessary artistic licence the genre demands, but both have made it clear that their political motivations shape their creative intentions. Mankell has said he began writing his Wallander novels, in which a world-weary detective battles with entrenched powers, as a response to the "xenophobia and racism" he saw in Sweden in the late 1980s. "The issues," he's said, "were always more important than Wallander himself."

And one of those issues was sexism. In this regard, it's not as if Mankell was a lone voice, ploughing the remote field of fiction. Even without Mankell's huge domestic and international success, the debate on these issues would have dominated Swedish cultural politics during the 1990s. And consequently, in 2000, a commercial sex act was passed that was seen at the time as a victory for radical feminism. It was made legal to sell sex, but illegal to buy it. In other words, criminality shifted from the prostitute to the punter, which in most cases meant from the woman to the man.

At the time, it was heralded as a major defeat for street prostitution and sex trafficking, and many countries, including Britain, have looked at copying the new Swedish model. In the wake of the law, the police had to refocus their attentions and also re-examine many of their attitudes in relation not just to prostitutes but to women in general. The most active and outspoken policeman in the battle for a less patriarchal perspective was, of course, Lindberg.

Many Swedes I spoke to suggested that Lindberg embodied a widespread cultural disconnection between official rhetoric and individual behaviour. As one well-placed observer of the Stockholm scene put it to me: "Some of the most outspoken male politicians on gender equality are also renowned as the most active pursuers of women."

But Gunnar Pettersson, a Swedish writer and commentator who lives in London, had a different take on the problem Lindberg represents. "Sweden has two elites," he told me. "The political elite is internationalist and neutralist in outlook, whereas the other elite, the military-industrial, is essentially nationalist and west-supporting. The two have left each other alone very largely, especially throughout the 20th century when the Swedish model was built up. The thing about Lindberg is that he adopted the rhetoric of the political elite but he belonged by nature and biology to the military-industrial elite, where these things are just horseshit. You just say it to get on in your career."

Whether Lindberg is a split personality or simply a flagrant opportunist is perhaps a question for psychiatrists to settle. What's arguably more significant is the hole his case exposes in the logic of political correctness. The theory behind the PC view of the world is that if you change the language, you change what the language describes, because perception alters reality: non-sexist expressions, for example, help to foster non-sexist thoughts. But what if the prescribed opinion is a false consensus? What if language is a disguise, a means of conformity that serves to conceal the underlying and more disturbing truth?

That would involve a novel variation on the longstanding Swedish preoccupation with deep-lying corruption. But not one that you'll find in the novels of Mankell or Larsson. Subtlety has never been either writer's strong suit, and some Swedes find their Manichean vision of Sweden rather limiting.

"I have always been suspicious and critical about people like Mankell and Larsson," says Lars Linder, "because I'm not a fan of this conspiracy theory. I'm an old leftist too, but I don't like when they pick out the old social democratic Sweden as paradise, and now the bad guys have taken over with all their hidden connections. It's simplistic and nostalgic. The kind of power abuse you see with Lindberg is much more interesting."

Mankell insists that The Troubled Man, published in Swedish last year and due to be published in English next year, is definitely his last Wallander novel. The plot once again features rightwing extremists as the antagonists. Sweden is renowned for its comprehensive social welfare, progressive liberalism and egalitarian spirit, and it's also consistently ranked by Transparency International as among the least corrupt nations in the world. So it seems perverse that when the country holds a mirror up to itself it so often sees female abuse, rightwing conspiracies and systemic corruption. Yet they remain emotive issues in Swedish culture.

A few days after Lindberg appeared in court, the employment minister, Sven Otto Littorin, tendered his resignation when he learned that a newspaper was about to run a story claiming he paid for sex with a prostitute four years ago. His unnamed accuser said she was inspired by the Lindberg case to try to prevent the powerful from escaping the consequences of their actions. He denies ever having paid for sex and the paper, Aftonbladet, offered no evidence, other than that the woman had seen Littorin on television and recognised him. And subsequently several observers have cast doubt on the woman's account, which is said to be filled with inconsistencies and inaccuracies.

But nonetheless Littorin resigned, citing press intrusion into his personal life. For the first time in decades Sweden found itself with a political sex scandal, something which the Swedes believed was a strange preserve of the British. In fact many observers find the Littorin saga more representative than the Lindberg case of the social changes under way in Sweden. For Petra Ostergren it marks a pronounced shift in Swedish public morals and illustrates how a narrow consensus has been effectively imposed. A feminist who is an outspoken critic of the commercial sex laws, Ostergren has been ostracised by many of her onetime allies in the women's movement.

"Fifty years ago Littorin would have had to resign if he was gay. Now we have not only criminalised the buying of sex but we've also stigmatised it to such an extent, he has to resign just because of the mere suspicion. Just as the gay man has been normalised, so the heterosexual buyer has been pathologised. To satisfy society's need for normality, you need something that is not normal. Now that is the sex buyer."

Naturally, that is not how many other feminists would see the situation. For them it is a matter of inequality and coercion. The sex worker, according to conventional intellectual wisdom, is in a weak position, socially and financially, and lacks power in any transaction with the consumer. Therefore she can't be said to be acting of her own free will, particularly, of course, if she has been trafficked and effectively held prisoner.

Ostergren counters that the vast majority of sex workers don't correspond to that description, and in any case forced and elective prostitution are entirely separate propositions. "We can distinguish between consensual and non-consensual or forced marriages," she says. "Why can't we make that distinction with prostitution?"

In answer to her own question, Ostergren outlines the questionable morality that informs some strategic social and political initiatives in Sweden. Fundamentally, she believes, what many Swedes dislike about prostitution is its transgressive, unhygienic, uncontrolled nature. She cites the substantial sterilisation programme overseen by the Social Democrats right up until the 1970s as evidence of an impulse among progressives to clean up and forcefully remove undesirable aspects of society.

"It's all part of the long project towards perfection and being modern," she says. "There is no room for drug addicts, prostitution or men who buy sex. It's an undercurrent of wanting to be a superior nation. We enjoy exporting that image. We love being on moral high ground."

The keys to Sweden, Kjell Nordström told me, are equality, modernity and consensus. A tall, bald professor of economics, Nordström is a kind of business guru who runs a consultancy on "funky capitalism". I visited him at his large apartment, worthy of a Wallpaper* magazine spread, on the leafy island of Djurgården that sits in the middle of Stockholm. It's a magnificent location whose panoramic views, it must be said, do not include the dark underbelly of fictional repute.

Nordström is another critic of the commercial sex law, on the practical basis that it doesn't work. According to some statistics, prostitution is almost back up to the level it was at when the law was introduced. But Nordström was also interested in a practical means of Swedes finding agreement on the issue.

"Conflict," he noted amiably, "is just not possible here. We've had 202 years of peace, and peace makes you a little bit weird." The inequality of prostitution, and therefore its backwardness, was what offended Swedes, he explained. To reach agreement on the issue, therefore, "You need to treat commercial sex in a very gender-neutral way."

I tried to imagine what that might involve, but I was defeated by the old-fashioned gender division of male and female. So Nordström spelt it out: "You have to have a whore house with men and women working alongside one another. You have to show that you've changed the concept to gain acceptance. People are not against sex here. It's a society where you can really talk about sex, it's easy to have sex with people. But you can't have exploitative sex because by definition you have used your power to buy another person. You owe an explanation on how it's not exploitative."

Unlike many Swedes, especially among the intellectual elite, Nordström does not believe that the Swedish project is floundering. He ran through a potted history of the economic miracle that powered the progressive reforms of the 20th century. In the 19th century Sweden was very poor and one in three of the population was an alcoholic. "We were a mini-Russia." In the 1920s a cradle-to-grave idea of social democracy was born in which an alliance between industrialists, unions and the state would produce universal social welfare. This was when the idea of the "people's home", a social democracy in which industrial wealth was redistributed for the communal good, first began to gain currency.

After the war, in which Sweden remained neutral, unoccupied and unbombed, it was one of the few countries in Europe with its manufacturing industry in full working order. Exporting everything from ball bearings to telephone exchanges, it rapidly embarked on a prolonged rise to prosperity. By the 1970s it was ranked as one of the three wealthiest nations in the world. As the money poured in, it was directed to building perhaps the world's most ambitious welfare system, with generous childcare, healthcare and pensions. In 1973, coinciding with a period of prohibitively high taxation, the oil crisis hit the economy. Three decades of growth ground to a halt and by the 1980s the government began to loosen its tight control on markets.

Rejuvenated, the economy expanded again but a disenchantment had entered the Swedish psyche, especially among the utopian left. The disparate doubts and grievances seemed to cohere with the killing of Olof Palme, which remains the defining event of postwar Swedish history. Its impact was bigger, relatively speaking, than the Kennedy assassination. Mankell once wrote a Wallander short story, entitled "The Pyramid", which examined the anxieties unleashed by Palme's murder, and Palme also turns up in The Troubled Man. Later this year, Mankell is also staging in Stockholm a play he has written about Palme, entitled Politik.

Palme was a curious figure. Born into an upper-class family, he assumed the clothes of modesty and frugality, yet at the same time retained a patrician sense of entitlement – he famously demanded that a ferry should return to port when he missed the last one on a trip to his holiday home. He was an internationalist who was fierce in his defence of Sweden's interests, and a neutralist who wooed the Soviet Union while discreetly favouring the west. He stood at the intersection of two different, and often contrary, strands of liberalism – the dual thrusts towards benign state intervention and increased personal liberty.

Walking home one night with his wife along Sveavägen – Stockholm's equivalent of, say, Piccadilly – Palme was killed by a mysterious gunman who vanished into the night. In the absence of a suspect, and incubated by a disastrous police investigation, a mass of conspiracy theories was hatched – some encouraged by the police – which fingered everyone from Kurdish gangsters to Saddam Hussein and the CIA.

Matters were not helped by the fact that the main witness – Palme's widow, Lisbet – refused to co-operate fully with the court, for reasons she has never explained. Her testimony led to the conviction of a violent street thug and alcoholic called Christer Pettersson. Pettersson had a previous conviction for murder for which, in a typically liberal piece of Swedish criminal justice, he had been sentenced to just six months in prison. He was sentenced to life, but was soon released when the judgment was overturned by the court of appeal.

The failure to apprehend the real culprit meant that Sweden's wound, or "national trauma" as it's often called, remained open for many years afterwards. Even now the scar tissue – the stubborn conspiracist paranoia – continues to impinge on various bones of political contention.

The most symbolic of these is the ongoing controversy over submarine incursions into Swedish waters during the 1980s. The provenance of the submarines that were known to hide off the coast of Sweden has been a subject of lengthy dispute. Much of the media believed they were Soviet vessels, while others suspected they belonged to Nato. Once again the troubling image recurs of something untoward lying beneath the smooth surface. Mankell is not alone in his opinion that these incursions, to which he refers in both The Troubled Man and Politik, amounted to a major national scandal. But if so then it may be Palme, the great hero of the left, who was at the centre of the embarrassment. There is growing evidence that some, if not all, of the incursions were Nato submarines, and persistent rumours in diplomatic circles that Palme knew of and agreed to their presence, as a means of affording protection from the Soviet Union.

Certainly Palme was a flexible politician when he needed to be, not least in the realm of sexual politics. Back in the 1970s news leaked out that his minister of justice, Lennart Geijer, was a major user of prostitutes. Although the information was accurate, as Palme knew, the prime minister strenuously denied the facts and the paper that published the story was forced to print an apology. Significantly, one of the villains in Mankell's novel Sidetracked is a minister of justice from the 1970s who is part of a sex ring that sexually and physically abuses women – much like Lindberg is accused of being. The character, who bears a resemblance to Geijer, is blamed for killing the idealism in Swedish politics. It will be interesting, therefore, to see Mankell's judgment of Geijer's boss, Palme, in his new play.

Kjell Nordström maintains that the nostalgia for the Palme era is a yearning to return to a simpler Sweden of greater state control. "There are people who miss the good old times when you could have a meeting, negotiate and then implement the decision. But we're no longer a small homogeneous country. We had to find other ways."

He also suspects that this harking back to a mythical golden age of integrity is partly a function of a Swedish male identity crisis. "Men are losing their position. Women have taken massive steps forward in the last 40 years. There are a number of areas today where it's difficult to be a man, where once there was a male language and now there are strong, powerful women, backed by law."

Lindberg's boss was a woman, he points out, and he was surrounded by women at work. "But," says Nordström, pouring me another glass of chilled wine, "he was not trained by the police university to exist and manage under these conditions."

That, in a nutshell, is the Swedish analysis that ultimately wins out over the conspiracist angst and liberal hand-wringing: here is a problem, let's establish better training and solve it. In many, perhaps most, ways it's an admirable attitude. After all, it bespeaks a progressive belief in the improvement, if not the perfectibility, of humanity. But such a pragmatic approach to problem-solving can also focus on the solution without really addressing the nature of the problem.

In this respect the Swedes who worry about the subterranean darkness might actually be on to something. It's just that they're looking in the wrong place. It's not necessarily in the system, or the state, or the police, or under the sea. It may just be in themselves. Whatever the reason Chief Lindberg may have been driving along with whips and handcuffs on his way to meet a teenage girl, the one certainty is that it was not because he lacked the appropriate training.