At a time of miserable conditions for the poor, sick and disabled people, the administration of the welfare state is a disaster. The grand projects the Department for Work and Pensions has launched since the general election have been bureaucratic fantasies and practical catastrophes. Ministers have wasted hundreds of millions of pounds of public money – Tory ministers, mark you, who pose as the defenders of hard-working taxpayers. For all that, Iain Duncan Smith tramps on without a thought of changing his ways: a character study in destructive pig-headedness.

At some level, he must know he is failing on all fronts. He and his state-sponsored propagandists pulsate with aggression. Anyone who tries to investigate his department is met with obfuscation and intimidation. Duncan Smith denounced the BBC for publishing a leaked memo, which showed that the costs of his employment and support allowance were growing at a formidable pace. The corporation was a more committed opponent of welfare reform than Labour, he cried (knowing how quickly the BBC folds under accusations of political bias).

Duncan Smith has targeted the Trussell Trust, an exemplary Anglican charity, which has mobilised the conscience of the nation and fed the hungry. He and his sly ministers suggested that visitors to food banks were freeloaders, rather than victims of poverty and the incompetence of Duncan Smith's department. As they did it, they were sitting on a government report, which showed the Trussell Trust was right. Low incomes and benefit delays were compelling hundreds of thousands of hungry people to beg for food as a "last resort", it said.

There is a journalistic scandal here. The Mail and the Telegraph attacked New Labour for its manipulation of the media, with considerable justification. But now their friends are playing the same tricks, where are the Paul Dacres and Peter Obornes defending honest reporting from governmental attack? The journalistic scandal hides the greater public scandal. Duncan Smith and his placemen have to intimidate because his department is the administrative equivalent of a failed state, a collapsed institution, where ministers mouth promises that never and can never come true.

Earlier this year, with barely concealed incredulity, Nicholas Wikeley, a judge at the Administrative Appeals Chamber, dismissed an attempt by Duncan Smith to keep secret a government report on the risk to public funds and public provision for the needy his vainglorious plans for universal credit could bring. He could see "no support" for Duncan Smith's argument that the electorate should know nothing about them.

Outsiders could see every reason why Duncan Smith would want to censor, however. Only a few thousand people are on a new credit that is meant to cover millions. Its computer systems have failed. About £140m has been thrown away and Margaret Hodge of the public accounts committee expects that many millions more will vanish. The DWP, she said, embarked on a £2.4bn project "with little idea how it was going to work".

It is not only the universal credit. If you think I am being too harsh, the Department for Work and Pensions annual report, published last week, said that Duncan Smith's Work Programme was "only helping one in 20 recipients of disability benefits find a job". The public accounts committee said Duncan's Smith personal independence payments scheme had been "rushed" through and the consequences for terminally ill and disabled people had been "shocking". Too often you see the sick and the ill-educated being told to log on to computers they don't have, to fill in forms they can't understand for IT systems that don't work.

Duncan Smith will not change. He is a neurotic authoritarian who wants to be powerful and expects to be obeyed, while living with the fear that everyone will dismiss him as a clown if he shows the smallest weakness. Those fears were amply realised in 2003. You may have forgotten that Duncan Smith was once leader of the Conservative party and saw himself as a future prime minister. Then his colleagues showed that, while he thought of himself as a statesman, they thought of him as an abject failure, a man who could not distinguish between reality and whatever ideological programme was animating his mind.

As Michael Gove wrote in the Times in the weeks before his downfall, whenever he heard Duncan Smith repeating the same tired slogans, without giving the slightest indication of self-doubt, Kipling's lines on know-nothing, learn-nothing stupidity came back to him. "The Dog returns to his Vomit and the Sow returns to her Mire, And the burnt Fool's bandaged finger goes wabbling back to the Fire."

By the time the Conservative party deposed him, Duncan Smith had barely an ally left. With singleminded fortitude, he rebuilt his career. He presents himself now as a great reforming minister rather than a prime minister, but his vices remain unchanged.

Labour politicians tell a story that captures both his vanity and his folly. A few years ago, Duncan Smith met Douglas Alexander, Rachel Reeves and Stephen Timms. He enthused about his belief in a universal credit that would merge taxes and benefits. He would free 6 million people from the poverty traps of welfare dependency and show them that work made them better off.

The Labour politicians admitted that universal credit was a fine idea. They had thought about implementing it many times. But you had to merge incompatible IT systems and find a way of updating the information on millions of people so that Whitehall knew almost instantaneously how much they were earning, what taxes they should pay and what benefits they should receive. Reforming a complex system would take years. If Duncan Smith rushed it he would be engaging in the vast and self-defeating social engineering the right accused the utopian left of forcing on the human race.

Duncan Smith would have none of it. The technicalities were trifles. All that was needed was the political will. And he, Iain Duncan Smith, the man of destiny, had the will to make it work. "We looked at him as if he was mad," one of the participants told me.

• On 6 July 2014, the Observer published the following in its For the Record column: "Why stubborn Iain Duncan Smith is no statesman" (Comment, last week, page 37) suggested that Richard Caseby was hired as director of communications for the Department for Work and Pensions by secretary of state Iain Duncan Smith. We accept that Mr Caseby was recruited to the senior civil service in an open competition overseen by the Civil Service Commission. He is not a political adviser. Apologies for any misunderstanding.