I was probably the most average chess player in the world. But there came a point where being average was no longer enough. I had become good enough to know how bad I was. I was attached to two clubs in south-west London, Surbiton and Kingston, and was mixing with players who were very good, who had international master titles, one notch below the coveted grandmaster title. I wanted to be like them. I wanted to feel at home on the 64 squares.

I was a middle-aged man who had done OK in life, but there was something missing. I hadn’t created anything substantial; hadn’t mastered a discipline. I craved substance, and saw in chess a possible way of laying down a marker. I would become an expert, demonstrate that I wasn’t just a dilettante. After a lifetime of chess mediocrity, I set out to achieve excellence, for the first time in my life to truly master a world, to become good – not just good at chess, but at living. To get really expert I would have to be focused, disciplined, ruthless even – all the things I had found it difficult to sustain in an often rackety life.

Truth, beauty and the art of success at chess | Letters Read more

My intermittent love affair with chess began when I was 11. I was certainly no prodigy. For one thing, I started too late – some 11-year-olds are virtually grandmaster standard. I was OK but, as with most other things in my life, didn’t work hard enough at the game. I wanted instant brilliancies; refused to do the slog of reading books, and saw chess as an art not a science. I had a friend who saw it as a science not an art, read books on opening theory, and always beat me. I played for my school, and had a decent record, but what I mainly remember is that we always got biscuits and orange juice before each match. Schools admired boys who played chess. Unlike the girls at the school, who tended to favour rugby players.

Membership Event: The Long Read live at the Hospital Club

As part of my quest, I thought it might make for a poetic experience to return to Newport, the place of my birth, to play in the South Wales summer congress. I was staying with my parents, who acted as if the intervening 38 years had never happened, making me sandwiches and giving me a Thermos flask of tea to see me through the rigours of the day. My first game was on Saturday morning. My opponent had a relatively low grade (grades in chess are not unlike golf handicaps, and designate how good players are: the top-rated player in the UK has a grade of around 280, mine was around 133) and he played like it, giving up a pawn for nothing in the opening, and then losing a piece. The game was effectively over, and it irritated me that he insisted on playing on. I don’t know if it was boredom, fury, tiredness or simple incompetence but, as we approached the endgame, I committed the mother of all mistakes, blundering away a rook.

He was in a position where he could endlessly check my king with his queen if he wished. He offered me a draw, trying to mask his relief and disbelief at the mistake that I had made, and I had to accept. I had thrown away an easy win and felt ridiculous. I fled the tournament hall, sat beside the murky waters of the River Usk, bolted my sandwiches, and tried not to burst into tears. This was the most abject failure I had yet had to endure, and it would be difficult to recover. So much for the poetry of my return to Wales.

I fled the hall, sat beside the murky waters of the River Usk, bolted my sandwiches, and tried not to burst into tears

In the middle of one of my later games at the Newport congress, I suddenly asked myself whether I was really enjoying playing. I had the same thought as I watched the deciding game in my section, in which a 17-year-old was playing a crop-haired, middle-aged man to decide who would take first place. Both were down to their last few seconds, the youngster was shaking, and almost every move they made was an error, a rook mislaid here, a bishop casually tossed away there. Eventually the crop-haired man lost on time – in tournament chess, moves have to be made in a set time limit – and the 17-year-old had his prize. But the youth looked drained, shell-shocked, incapable of feeling any pleasure at his victory. Why did we put ourselves through this?

I had had similar doubts after that earlier game in Newport when I had thrown away a rook with what may have been the most ridiculous move in the history of chess. Why was I here making a fool of myself? What was the point? John Saunders, a former Welsh international, was giving me occasional coaching as I tried to crack the chess code. But he was also good at offering homilies. “Playing chess is a vale of tears,” he said during one of our training sessions as we examined an especially egregious, error-riddled game I had played. “This teaches us a very valuable lesson. It doesn’t matter what you do, how you play or how you change your approach to the game. Chess is just a bitch that bites you in the arse.”

Saunders was expressing, albeit less elegantly, a view propounded by HG Wells in an essay entitled Concerning Chess in 1901. “The passion for playing chess is one of the most unaccountable in the world,” wrote Wells, who at times shared that passion but never appears to have become very adept at the game. “It is the most absorbing of occupations, the least satisfying of desires, an aimless excrescence upon life. It annihilates a man.”

Does the pain always outweigh the pleasure? Siegbert Tarrasch, the best player in the world (though never world champion) in the 1890s, thought not. “Chess, like love, like music, has the power to make men happy,” he wrote in the preface to his book The Game of Chess, published in 1931, three years before his death. But is that really true? A blog by the English grandmaster Danny Gormally encapsulated the nightmares chess can induce. “I played a game a couple of days ago where I lost from a rook up,” wrote Gormally. “At the end of the game I completely lost my rag and started screaming, which was quite embarrassing in hindsight. Really I was angry at myself. Other players in the tournament, including my opponent, saw this meltdown and must have thought, ‘What a nutter’. Increasingly I’m suffering from poor emotional control. You’d think I’d get calmer as I get older, but at times I even feel I’m teetering on the brink of madness.”

The Dutch grandmaster Hein Donner also captured the pain and paranoia that chess can induce. Writing about the great world championship match in 1972 between Bobby Fischer and Boris Spassky, and the beginnings of Spassky’s disintegration, he poured scorn on the idea that chess was one of the least demanding of pursuits. “What comes dramatically to the fore in this match,” he wrote, “is that chess is a tough sport. What seems so easy at first sight in fact puts a greater pressure on the players than any other branch of (physical) sports. To sit immovably still for five hours on end, in a condition of semi-consciousness, under the heavy burden of a possible mistake – all this opens the door wide to serious distress.”

Facebook Twitter Pinterest Bobby Fischer, right, and Boris Spassky contest the world championship in Reykjavik, Iceland, in 1972. Photograph: J. Walter Green/AP

A chess win offers a high that is almost sexual – that moment of release after sitting at the board for four hours plus. But do the highs make up for the lows? John Saunders, my chess coach, didn’t think so. “The pain of defeat is greater than the joy of victory, at least for a pessimist like me,” he insisted, which may be one reason why he retired from competitive play in his late 40s to concentrate on writing about the game. He found it even harder to derive satisfaction from playing once computer chess programs, now available to everyone, became all-seeing and able to spot flaws even in games you thought you had won well.

Why do we put ourselves through it? It seemed to be a question that concerned amateurs more than the pros, who had played since they were five and for whom it had become a way of life, as natural as breathing. Some eulogised upon the game’s beauty and heritage, but for most it was enough that they excelled at it.

David Spanier, another journalist (and hapless amateur) who fell in love with chess and wrote a book about it 30 years ago, had a simple explanation for the game’s enduring appeal: “Chess is a substitute for life itself ... Over the board all the dramas and colours of living are continually being played out in imagination. [It has] something like the effect of a gently powerful, pervasively consuming, hallucinatory drug.” Robert Desjarlais, a chess-loving American anthropologist, also uses the analogy of drug addiction in his book Counterplay: “Just as the chemical properties of heroin directly and immediately affect the central nervous system, so chess can lock into certain pathways of the mind, and it doesn’t easily let go.”

Desjarlais, though, doesn’t stop there. He examines this peculiar passion, the reasons we become addicted. “For some, chess is a hobby picked up along the way,” he writes, “while for others it’s a cathedral of truth and beauty. The attractions often relate to the drama that each game promises, the competitive challenge in pitting one’s skills against another’s, the intricate complexity that comes with any chess position, the rewarding intellectual conversation that takes place between two minds during a game, how focused concentration can take a person into a domain of pure thought removed from the hassles of everyday life, the way chess enables people to know their mind better, the pleasures of learning and participating in the conceptual history of modern chess, the camaraderie to be found at chess clubs, the thrill of accomplishing something creative at the board, and the way in which truth and beauty – and perhaps a measure of wisdom – can be found in chess.”

I liked that truth and beauty came at the end of Desjarlais’s list. When I set out on my mission, I wanted to grasp the truth of a position and create something beautiful. But as I played in more and more tournaments, I just wanted to win: to beat my opponent and improve my grade. Of course one does that by understanding more about chess, by becoming more adept at analysing positions, by getting closer to their mystical “truth”. But I had also come to see that strength of character, calmness under pressure, the sheer will to win were just as important. Maybe even more important at the amateur level, where errors abound. “The winner of the game is the player who makes the next-to-last mistake,” said Savielly Tartakower, a Polish grandmaster from the first half of the 20th century and the game’s greatest aphorist. Kill or be killed. To hell with truth.

The romantics would disagree. In the early decades of the 20th century, chess was seen as a beautiful, rarefied pursuit akin to philosophy and the arts. The French artist Marcel Duchamp loved the game so much there were rumours that he planned to forgo art to concentrate on chess. “While all artists are not chess players,” he said, “all chess players are artists.”

Duchamp was an excellent player, good enough to play for France in chess olympiads and to get a draw in a tournament in 1929 against Tartakower, a result that pleased him so much that he framed his scoresheet. There is, though, a conflict at the heart of chess, one that even Duchamp, who did not disguise the inherent violence of the game, recognised. Chess aspires to the condition of art, with beautiful ideas and aesthetically pleasing tactical combinations, but it is also a base struggle to destroy the other player.

Facebook Twitter Pinterest Marcel Duchamp: ‘While all artists are not chess players,” he said, “all chess players are artists.’ Photograph: Eliot Elisofon/Time & Life Pictures/Getty Image

Hein Donner addressed the question of whether chess was a profound endeavour or a complete waste of time in a newspaper column published in 1959. He imagined world champions, once they had reached the summit, asking themselves: “What purpose has all this energy served? Has there been a point in all this strenuous effort?” He went on to dismiss the then world champion Mikhail Botvinnik’s rationalisation of chess as a branch of art. “A chess player produces nothing, creates nothing,” argued Donner. “He only has one aim: the destruction of his opponent. This may be done in a very artistic way. But there is something strange about those perfect games in which deep strategies or brilliant combinations secure victory. They are published all over the world and are included in textbooks, these games that are ‘all of a piece’, but in fact they are not chess games at all, they are monologues. A real chess game can only be experienced by two people. Nothing can be said about it. Nothing comes of it.”

According to Donner: “The whole point of the game [is] to prevent an artistic performance.” The former world champion Garry Kasparov makes the same point. “The highest art of the chess player,” he says, “lies in not allowing your opponent to show you what he can do.” Always the other player is there trying to wreck your masterpiece. Chess, Donner insists, is a struggle, a fight to the death. “When one of the two players has imposed his will on the other and can at last begin to be freely creative, the game is over. That is the moment when, among masters, the opponent resigns. That is why chess is not art. No, chess cannot be compared with anything. Many things can be compared with chess, but chess is only chess.”

Did I want to win at all costs, or was I trying to serve some higher calling? My view oscillated wildly during the course of my odyssey. Occasionally I did experience the joy of trying to capture the truth – I remember one out-of-body experience in a tournament at Hastings where time seemed to freeze and nothing else mattered but burying myself in the position in front of me, questioning its meaning, divining what it really meant. But at other times, I didn’t give a damn about the meaning – or, indeed, the means. All that mattered was the end of beating my opponent, crushing him into the dust.

A year or so into my three-year mission, I was in one of my “truth and beauty” periods. Treat chess with respect, play for the right reasons and glory will follow. Inevitably, in my next game, it didn’t work out that way. I had agreed to play a game for Kingston, one of my two league teams, against Redhill. I’d been playing for Kingston in a desultory way for several years without ever really feeling attached to the club.

Whereas my other club, Surbiton, met in the congenial surroundings of a large, rambling detached house that doubled as a day centre for the elderly and was doing well, Kingston stumbled from crisis to crisis. It was short of players and, having been evicted from a Quaker hall that was being redeveloped, now had no real home and was forced to resort to an Asda in Roehampton – a world away from the chandeliered Viennese cafes in which chess had flourished in the early part of the 20th century. The romantic in me hankered after that golden-age ambience, and Kingston’s plight seemed to sum up the decline of chess since it had been front-page news in the Fischer era.

These little London clubs – replicated in towns all over the UK, where chess aficionados meet in the back rooms of pubs and social clubs – are hangovers from a time when the game was thriving. Back then, there was little else to do, and men – it was invariably men – were looking for a way to fill dark winter evenings. Chess, like fishing and pigeon fancying, is a great form of structured time-wasting.

Kingston had 15 or so regular players, all but one of whom were men. Mostly, they were past 50 and had played chess all their lives. They had grown up in that postwar period when it was part of their school life, and the game had then become an integral part of their adulthood too. Even the ones who took breaks to marry and have children, as I had, eventually came back. Chess had them for life, and for some the board was their world, perhaps even their salvation.

We were playing in Asda’s cafe, thankfully closed to the paying public – the clatter of cups and a gawping audience would have been an indignity too far. I was up against a player graded 160 who annoyed me with his habit of invading my space by stretching his legs out under the table. He played slowly and fell hopelessly behind on time. I was gradually building a better position and had a huge time advantage, so as we neared the time control – we each had to play our first 35 moves in an hour and a quarter – I felt it was inevitable I would win. Shades of Newport, and another demonstration of Donner’s adage that the position he hated most was one that was “totally winning”.

Foolishly, I forgot this sage observation and just waited for my opponent to crumble. But as the time control approached, he started to speed up, banging his hand down on his clock and stopping recording his moves, as is sometimes allowed when you have less than five minutes left. I was still playing at a leisurely speed, convinced I had more than enough time to secure victory. My advantage was getting bigger, and I felt sure he would resign at any moment. But he didn’t resign: he carried on, played ever wilder moves and set me a series of problems that ate into my time. I lost control of the position and, worse, lost track of how long I had left, so that the flag on my clock fell as I was about to make my 35th move. I had lost on time – a game I had spent the past hour believing I had won. He had somehow made 10 moves in a minute and suckered me into defeat. I was devastated, hardly able to speak or think coherently.

The chess world championship was taking place when I played this appalling game. The young Norwegian Magnus Carlsen was giving the reigning champion Viswanathan Anand a beating, and the latter had just lost a crucial game with a terrible blunder that virtually guaranteed Carlsen would become the title holder – at 22 the joint youngest player, along with Garry Kasparov, to become world champion. After Anand’s defeat, grandmaster and former British champion Jonathan Rowson tweeted: “Just as men will never ‘get’ what it feels like to give birth, non-chess players will never know the unique torment and anguish that flow from such a defeat.”

Facebook Twitter Pinterest Magnus Carlsen, right, playing Vishwanath Anand in the world chess championship in 2013. Photograph: Manjunath Kiran/AFP/Getty Images

Rowson had summed up precisely the way I felt after losing the Asda game. I believed that somehow I had been cheated. I deserved to win that game. My opponent may have been nominally a 160, but I had played the better chess. Natural justice demanded that I should be the victor. I wanted to make some sort of protest. Did he stop recording his moves too soon? Did the five-minute rule really apply in these circumstances? Should we have made an effort to reconstruct the game afterwards – neither of us was scoring by the end – rather than offering a frosty (on my part) handshake and disappearing into the night?

I made myself look ridiculous by moaning about the injustice to anyone who would listen. I should have taken it on the chin, but I seemed incapable of doing so. I had a terrible, sleepless 24 hours, until gradually some sort of sanity reasserted itself. A few days after my defeat, while commentating on a rugby match in which Wales once again fell short against South Africa, the former England hooker Brian Moore said: “In sport you don’t get what you deserve.”

When the new mid-year grades were announced about halfway through my quest, I had managed to add a few points – pulling myself up painfully to 136, an old codger’s grade if ever there was one. I was about to “celebrate” my 57th birthday – an age to consider retirement, not to set yourself improbable challenges. John Saunders liked to see the battle for chess excellence as a mountain to be climbed. I felt I was still struggling to find a route up from base camp. How on earth could I begin the ascent?

I had attached great importance to winning the Felce Cup – the annual championship for Surrey players graded under 140. I had somehow managed to make it to the final, against a likable Frenchman who played for Wimbledon. This was played over two games so we could reverse colours – playing with the white pieces confers a small advantage because you get first go and can, in theory, dictate the play. I won the first game with white, so all I needed in the second game to win the cup was a draw. As the big day approached, I told myself I needed to stay relaxed: if I wanted this too much and put pressure on myself, the likelihood was I would lose. Relaxed concentration – that’s what was required; an intense focus on the game, but sufficient mental flexibility to avoid freezing and to allow myself to know when to strike.

I took the day of the final game off, stayed in bed late and rested ahead of the evening encounter. Despite playing with the black pieces, I managed to get a small advantage by move 15. This seemed to provoke him, and he launched an unwise and premature thrust with his pawns. This was not his natural game – I had played him before and knew he was happier waiting for me to overreach myself – but he needed to win to force a play-off. He sacrificed a pawn for a speculative attack, I parried and launched a counter-attack, he failed to see how potent it was and, on move 31, faced with a simple three-move sequence that would force checkmate, he resigned. I was champion of the great, historic county of Surrey (brackets, division three). Now, at last, I had a large cup to fondle, polish, cherish.

After winning the Felce I felt wonderful, but there was still part of me that asked why I was doing this

After winning the Felce I felt wonderful, but there was still part of me that asked why I was doing this. Was chess really a pursuit worth wasting your life for? Was it a boon or a curse? The American writer, inventor and statesman Benjamin Franklin, a keen (though by all accounts not very good) player, believed it was the former, producing an essay titled “The Morals of Chess” (published in 1786 but written half a century earlier) which argued that it was good for the soul.

“The Game of Chess,” wrote Franklin, “is not merely an idle amusement. Several very valuable qualities of the mind, useful in the course of human life, are to be acquired or strengthened by it, so as to become habits, ready on all occasions. For life is a kind of chess, in which we have often points to gain, and competitors or adversaries to contend with, and in which there is a vast variety of good and ill events that are, in some degree, the effects of prudence or the want of it.” Franklin said chess taught you foresight, circumspection, caution and optimism.

Facebook Twitter Pinterest Contestants in the Four Nations Chess League tournament at Holiday Inn, Birmingham, in 2015. Photograph: John Robertson/The Guardian

More than two centuries later, former world champion Garry Kasparov wrote a self-help book based on the belief that chess offers signposts for life, titled How Life Imitates Chess. The game, he argued, was an “ideal instrument” for developing effective decision-making. “Psychology and intuition affect every aspect of our decisions and our results,” wrote Kasparov. “We must develop our ability to see the big picture, and deal with and learn from crises. Such decisive moments are turning points ... we select a fork in the road knowing we won’t be able to backtrack. We live for these moments and in turn they define our lives. We learn who we are and what truly matters to us.”

“Chess gave me a way to channel difficult emotions into something creative and constructive,” said Jonathan Rowson, a brilliantly insightful writer on the game, in a column in the Glasgow Herald. “From the age of about five to about 25, whatever sadness and confusion I held inside me became fuel in my quest to improve.” Rowson was expressing the reason many people play the game. It was a source of validation, a place where the rules and rituals made sense.

At the outset of my odyssey, I had hoped chess would be good for me – that it would make me a better, more complete, more focused person, able in some way to engage with life in a less superficial manner than in my first five decades. I had largely been disappointed. I had become a marginally better chess player, but there was no great evidence I had become a better, more focused person. I was still overweight – my dietary and gym regime, designed to back up my assault on this chess Everest, had never quite clicked into place; my commitment to work, in both chess and my rather understated career, remained uneven; I still felt I was underachieving and too inclined to drift along.

Chess was supposed to change all that – to make me a more driven, purposeful individual, and teach me the life lessons espoused by Kasparov. Some hope! Siegbert Tarrasch said “chess has the power to make men happy”, but I thought that the former British champion James Plaskett’s observation that “the pure and solitary nature of chess attracts some fragile minds and helps hold them together” was nearer the mark.

According to his biographer Calvin Tomkins, Marcel Duchamp knew why he loved chess and why it sustained him all his life. “Duchamp’s working methods were marked by an almost mathematical precision,” wrote Tomkins, “and one of the things he loved about chess was that its most brilliant innovations took place within a framework of strict and unbendable rules. ‘Chess is a marvellous piece of Cartesianism,’ he once told me, ‘and so imaginative that it doesn’t even look Cartesian at first. The beautiful combinations that chess players invent – you don’t see them coming, but afterward there is no mystery – it’s a purely logical conclusion.’”

Infinite variety and possibility within a strict framework was one of the joys of chess. It had become apparent to me why Duchamp likened chess players to artists. It was not so much that they created art, more that they strove to enjoy the mental freedom of the artist and give themselves scope to create. It was an attitude of mind. “I have the life of a restaurant waiter,” Marcel Duchamp once told the art critic Pierre Cabanne, describing his free and easy lifestyle. But he could just as easily have said “I have the life of a professional chess player” – freed from the workaday world and living in a constructed reality, a limitless imaginative landscape made up of 64 squares.

As the end of my journey approached, I had come to believe that chess, whatever Franklin and his followers might have thought, was not a moral good or a guide to living. Like much of art, which it resembled without being able to embody, it was an assault on bourgeois convention, promoting play above “real” life and engaging in conflicts that mimic the world beyond the board, while at the same time mocking it. Why shouldn’t grown men and women play for a living, or, in the case of many hopelessly impoverished chess professionals, a pseudo-living? Chess might not make you a better person, but it could make you a freer one.

Main photograph: Stephen Moss playing Magnus Carlsen in 2009. Credit: Linda Nylind

The Rookie by Stephen Moss (Bloomsbury Publishing, £18.99). To order a copy for £15.57, go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min. p&p of £1.99.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.