The slowness of the Metropolitan police and the then director of public prosecutions, Keir Starmer, to prosecute over widespread voicemail hacking was feeble at best. Those prosecutions needed to happen, but that does not mean Andy Coulson or his journalist colleagues should now be in prison.

The custodial sentences are ridiculous; they serve no public purpose. The conviction itself will be the most severe part of Coulson's punishment. If he should make amends, it would surely have been better to work for a worthy cause than cost the taxpayer nearly £40,000 a year to bang him up. Community service or teaching adult literacy somehow seem right.

Coulson's sentence tells us more about the vindictive nature of our justice system – and of public opinion – than it does about his crimes. In a century, we will look back on today's penal practices with scarcely less surprise than the way we currently see, say, the 1723 Black Act, which introduced 50 new hanging offences, including one for "hiding in a forest while disguised".

The Old Testament injunction to take an eye for an eye, which today sounds so bloodthirsty, was in fact an appeal for proportion where death or dismemberment were meted out for trivial offences. As a reforming measure to the Black Act, petty thieves were later spared the gallows and transported to Australia instead.

Coulson was, for his four years as one of Rupert Murdoch's capi, a member of that informal legislative body which has more pernicious influence over public policy than any other, namely the committee of tabloid newspaper editors. He was utterly conventional, calling for tougher sentences and berating "holiday camp" prisons. His incarceration, if it makes his red top friends think twice, will have one small consolation.

The tabloids, though, merely mirror opinion. They make money by giving people what they want, however vile. Schadenfreude – the delight in others' misfortunes – is a common human emotion. The evidence of our criminal justice system, and our tabloids, suggests there is something in the British psyche that likes to see others suffer.

Are we much different today to the 30,000 people who gathered in 1849 to witness the public hanging of a husband and wife at Horsemonger Lane prison, Southwark? Charles Dickens, ever the reforming voyeur, wrote: "The horrors of the gibbet and of the crime which brought the wretched murderers to it faded in my mind before the atrocious bearing, looks, and language of the assembled spectators."

If your life is hard, seeing someone worse off than you appears horribly to lift the spirits. Prison is the new public execution, with the tabloids scrambling to take pictures of any imprisoned celebrity and to invent ludicrous stories about their humiliations inside (as I can testify from my own experience).

Of course, no substantial society has yet survived without prison. Locking up dangerous people makes society safer. But who? Given the gravity of his crimes, and the suffering of innocents, custody must be right for Rolf Harris. But surely not for Coulson and many others on short sentences.

The general proposition that more prison means less crime is suspect. Internationally, crime has fallen with falling prison populations, and has risen with rising ones. Reoffending following community sentences is lower than reoffending following a short prison sentence. Prison is a college for spreading best criminal practice. We should keep it for serious offences and serial offenders.

We are thankfully less punitive than the United States, which holds the world record for prisoners (relative to population). But at least in the US, as a country still motivated by Christian values, there is a willingness to forgive. What of former drug abuser, alcoholic and ne'er do well George W Bush? He found the Lord, so make him president.

In Britain, we have the punitive instincts of the Americans without the redeeming compassion. Given that 30% of English and Welsh men have a criminal conviction by the age of 40, this is both hardship and waste. England and Wales are the punishment champions of western Europe. We lock up 149 of every 100,000 people, compared with 103 in France and 78 in Germany. If we imprisoned as much as Sweden, we would have 35,000 prisoners not 85,684. This would save nearly £2bn a year, enough to fund better community sentences and probation.

It is not just a question of more imagination over short sentences. A life sentence now means 12-14 years in prison, not the previous nine. Yet people who kill inflamed by passion are usually tortured by remorse and rarely repeat their crime. Why not parole them early? There are also nearly 6,000 prisoners on indeterminate sentences dished out under Labour. If it was right to abolish such sentences in 2012, it is also right to accelerate parole for those sentenced as such in the past.

Half of Coulson's time will come off unless he reoffends. He is then likely to be on an electronic tag for half the remainder, so he will have served four and a half months of his 18 months. Still too long, but it would surely be better not to indulge in a pantomime of legally sanctioned lying about the length of sentences, always pounced on by the tabloids. By failing to confront our ghouls – and their tabloid harpies – we merely let them haunt us again.