Illustration by Ralph Steadman

The last time I remember going out with my mother it was to Hampstead Heath. We drove there in my car, then walked arm in arm along the terrace in front of Kenwood House. As if elliptically commenting on our own halting progress my mother said: “The good thing about being a pessi­mist is that you’re never really wrong-footed; even before you’ve put one foot in front of the other you suspect that you’re likely to trip up, and that makes adversity much easier to deal with.”

She died three weeks later, lying in a bed in the Royal Ear Hospital. Not, you understand, that there was anything in particular wrong with her hearing; rather, despite the cancer that had metastasised from lymph to liver to brain, she remained highly attuned to the vapidity of yeasayers.

Indeed, I imagine the last thing she heard – and silently dismissed – before she slid into the coal-hole of inexistence was some well-meaning health professional or other, telling her it was all going to be all right.

That was a quarter-century ago, but my mother’s valedictory wisdom has stayed with me, informing my life, refining an epicurean attitude to personal life and a stoical one towards public events. For those who would dismiss pessimism out of hand, seeing it as a negative and self-fulfilling prophecy, let’s lay our jokers on the table right now: in respect of which of the major social and political developments of the past 25 years would optimism have been an appropriate attitude to take? My mother would have had to hang on only a few months in order to prop herself up in bed, or possibly lie supine, while I read aloud to her Francis Fukuyama’s essay in the National Interest “The End of History?”. How she would have snorted derisively at Fukuyama’s assertion that the end of the cold war would be followed by the worldwide dissemination of benign western liberal democracy.

Of course, in 1989, the immiseration of the former Soviet Union was just a gleam in wannabe oligarchs’ eyes and the rise of Putin’s Potemkin democracy lay some way ahead. The US was about to disengage itself from a range of proxy wars across the globe, in order to reinvest its peace dividend in the prosecution of a brand new range of hegemonic interventions.

A decade had already passed since the Camp David accords that were to have ushered in a peaceful era – but there was no sign of a lasting peace in the Middle East then and there certainly isn’t now. Indeed, US support of the Israeli state’s expansionist territorial aims remains to this day the festering pressure sore on the posterior of international relations. Mum, a Jew who believed passionately in justice for the Palestinian people, wouldn’t have been in the least bit surprised about this.

Nor, I imagine, would she have kept her sunny side uppermost as the western coalition’s air forces vaporised the retreating Iraqi conscript army at the end of the first Gulf war. An optimist, of necessity, believes in a future typified by knowns, because if – in the rousing chorus of the Blair government’s accession anthem of 1997 – “things can only get better”, then this must be in comparison with what already obtains. The pessi­mist, by contrast, is fully attuned to Donald Rumsfeld’s unknown unknowns: the black swans that swoop down out of a clear blue sky to annihilate thousands of New York office workers. The pessimist does not sanction foreign wars on the basis that democracy can issue forth from the barrel of a gun – which is not to say that pessimists don’t believe in the need to defend democratic values. Indeed, the chief paradox of whatever still obtains in the way of British greatness is that it derives from Churchillian pessimism: while the appeasers were optimistically waving their brollies, Winnie was scrying the storm clouds of the Nazi blitzkrieg. It was when optimism got the better of him – believing in the continuation of British rule in India – that Churchill’s Pollyanna intransigence contributed to the deaths of up to three million Bengalis in the 1943 famine.

No, in foreign affairs a healthy dose of pessimism – if by this is meant a willingness to accept that things may be for the worst in a less-than-perfect world – is definitely indicated. But domestically the optimism (if you can call it that) of the Thatcher-Blair neoliberal consensus hardly seems to have been borne out. Abed in the late 1980s, receiving cancer treatment that may have been less advanced than that of today, but which nonetheless was administered free and on demand with no caveats, my moribund mother would have undoubtedly been right in taking a gloomy view of the sell-off of our public assets.

As we shiver our way through an interminable winter, facing both fuel poverty in individual households and collective energy insecurity after having bartered our oil reserves for a mess of banker’s pottage, the much-vaunted efficiency of the market seems like just another optimistic mirage. Food banks opening at the rate of two a week, sickness benefit claimants about to be struck off by private contractors without any recourse to justice – these are developments that wouldn’t have fazed her.

She regarded the auto-cannibalistic tendencies of capitalism not from a theoretical perspective, but with the weary eyes of an American child of the Depression era. She had witnessed her own father keep the family afloat by organising fire sales for bust department stores.

Indeed, what are speculative bubbles if not the purest example of optimism run wild? The same sort of loony thinking that once invested in perpet­ual motion machines leads the contemporary credulous to believe that financial wizardry can conjure something out of nothing. The same glad-eyed and groundless enthusiasm for the Good News that the Redeemer’s arrival is imminent also leads people to believe that economies can continue to grow for all eternity, spawning more goods for more clap-happy consumers. I’m by no means the most eminent Cassandra to have pointed out that there’s a worm in the Enlightenment’s apple of knowledge; this distinction belongs to my New Statesman colleague John Gray. But, by contrast with him, I retain the same ideals as I always did: a belief that an egalitarian and essentially socialistic society is worth striving for.

Is this a paradox? I think not. Moreover, I also hold that a healthy streak of pessi­mism is pretty much mandated by such idealism. Let me explain. For every instance of a pessimistic forecast being fulfilled that I’ve set out above, the optimist can probably instance a counter-example. So be it. But the optimist also thinks that it is her willingness to entertain a better future that acts as a psychic midwife to its birth. How, the optimist argues, can you be bothered to struggle for a state of affairs that you regard as at best unlikely, and quite possibly altogether unattainable? The answer lies in the appreciation that the political and the personal are linked not instrumentally, but existentially. Subscribing to an ideology, whether it bases its appeal in the reasonable prolegomena of a Rousseau-inflected state-of-nature, or one of the instinctive and Hobbesian variety, nonetheless involves the individual in an act of deferral of the form: not now, but given such-and-such, then.

It is this “such-and-such” that forms the basis of all institutionalised appeals to political action: the communist utopia is forestalled quite as much as the thousand-year Reich; both retreat in advance of the measured tramping of the mobilised masses. At a less dramatic level, politicians in our highly imperfect (but still vaguely operable) representative democracies exhort us with their manifesto promises of jam tom­orrow and seek to remind us of the jam we spread on yesterday’s bread. It’s no wonder that electorates that are gummed up within the mechanisms of internet commerce find such appeals increasingly difficult to hear above the whine of their computers. After all, this is the most compelling contemporary paradigm of gratification: push the button to receive jam by express 24-hour delivery; and if you sign up for repeat deliveries, you can indeed have tomorrow’s jam today.

It is this consumerist ethic – if it can be so glorified – that has eaten away at any remaining semblance of altruism, its chomping in synchrony with the optimistic belief in the power of the market to unite mouths efficiently with jam. And this also explains why all political parties and charitable organisations now aspire to the form of commercial enterprises, complete with marketing departments and tax breaks for donations. Implicit in all of these activities, whether ostensibly dedicated to social welfare or to capital aggregation, is a utilitarian calculus. The nature of the good – or goods – may be disputed, but the conviction remains that it can be factually accounted for and numerically arrived at.

Yet to live a full life is not to cede such a large percentage of it to a purely statistical perspective; such a life – to borrow the title of Céline’s novel – is merely death on the instalment plan. And it is the optimist, para­doxically, who enforces such a life on the generality of humankind with her plea that we look to a better future.

I have, as you have probably realised, a good deal of sympathy for that apocalyptic tendency that led Spanish anarchists to burn the town hall records and string up the priest. But I don’t think we have to resort to such excesses in order to reclaim the primacy of the here, the now and the individual over the insistent compulsions of the there, the then and the collective. All that’s necessary is to expect the worst but live hopefully, if by “living hopefully” is meant to invest the present in the raiment of all the idealism any of us could wish for – to practise, in the telling phrase of Basho, the Japanese Zen poet, “random acts of senseless generosity”.

We do not arrive at any idea of what is best for the collective unless we are prepared to seize the day and practise it on our own behalf. Most mature individuals understand what this means in respect of themselves – it’s just all those feckless others that they don’t trust to act appropriately. And so, by one means or another, they seek to organise society in such a way as to corral the human kine and herd them towards pastures new. But really, the sweet-smelling grass is beneath our hoofs right now: what is required is that we take pleasure in what is available to us. I said above that my pessimism resulted in an epicureanism when it came to personal life. Unfortunately, in our gastro-fixated culture, the epicurean is associated with fancy concoctions of wheatgrass, rather than the stuff growing close to hand. We need to redress this balance and understand that once the basic necessities of life are accounted for, all the rest can be creative and even wilful.

The optimist can never embrace this perspective, driven as she is by an inchoate need that can always be shaped by others so as to tantalise her. The optimist – again, paradoxically – lives in fear of a future that she endeavours, futilely, to control. The optimists can never be that most desirable of things: a meliorist, because every setback is necessarily a disaster. For the pessimist, it’s simply a matter of shit happens, but until it does, make hay.

But if I may end, as I began, on a personal note (rightly so, given the tenor of what I’ve had to say), while I have maintained a pessimistic cast of mind for the past quarter-century, there are many areas of my life in which my pessimism has been unwarranted – none more so than in respect of the New Statesman. In 1988 my career as a contributor was in distinct abeyance; 25 years on it’s going strong, and long may it and this vehicle for it continue.

Will Self’s most recent novel, the Booker-nominated “Umbrella”, is published by Bloomsbury (£18.99). He writes a weekly column for the New Statesman