Jack White, the former frontman of the White Stripes and an influential figure among fellow musicians, likes to make things difficult for himself. He uses cheap guitars that won’t stay in shape or in tune. When performing, he positions his instruments in a way that is deliberately inconvenient, so that switching from guitar to organ mid-song involves a mad dash across the stage. Why? Because he’s on the run from what he describes as a disease that preys on every artist: “ease of use”. When making music gets too easy, says White, it becomes harder to make it sing.

It’s an odd thought. Why would anyone make their work more difficult than it already is? Yet we know that difficulty can pay unexpected dividends. In 1966, soon after the Beatles had finished work on “Rubber Soul”, Paul McCartney looked into the possibility of going to America to record their next album. The equipment in American studios was more advanced than anything in Britain, which had led the Beatles’ great rivals, the Rolling Stones, to make their latest album, “Aftermath”, in Los Angeles. McCartney found that EMI ’s contractual clauses made it prohibitively expensive to follow suit, and the Beatles had to make do with the primitive technology of Abbey Road.

Lucky for us. Over the next two years they made their most groundbreaking work, turning the recording studio into a magical instrument of its own. Precisely because they were working with old-fashioned machines, George Martin and his team of engineers were forced to apply every ounce of their ingenuity to solve the problems posed to them by Lennon and McCartney. Songs like “Tomorrow Never Knows”, “Strawberry Fields Forever”, and “A Day in the Life” featured revolutionary aural effects that dazzled and mystified Martin’s American counterparts.

Sometimes it’s only when a difficulty is removed that we realise what it was doing for us. For more than two decades, starting in the 1960s, the poet Ted Hughes sat on the judging panel of an annual poetry competition for British schoolchildren. During the 1980s he noticed an increasing number of long poems among the submissions, with some running to 70 or 80 pages. These poems were verbally inventive and fluent, but also “strangely boring”. After making inquiries Hughes discovered that they were being composed on computers, then just finding their way into British homes.

You might have thought any tool which enables a writer to get words on to the page would be an advantage. But there may be a cost to such facility. In an interview with the Paris Review Hughes speculated that when a person puts pen to paper, “you meet the terrible resistance of what happened your first year at it, when you couldn’t write at all”. As the brain attempts to force the unsteady hand to do its bidding, the tension between the two results in a more compressed, psychologically denser expression. Remove that resistance and you are more likely to produce a 70-page ramble. There is even some support for Hughes’s hypothesis from modern neuroscience: a study carried out by Professor Virginia Berninger at the University of Washington found that handwriting activated more of the brain than keyboard writing, including areas responsible for thinking and memory.

Our brains respond better to difficulty than we imagine. In schools, teachers and pupils alike often assume that if a concept has been easy to learn, then the lesson has been successful. But numerous studies have now found that when classroom material is made harder to absorb, pupils retain more of it over the long term, and understand it on a deeper level. Robert Bjork, of the University of California, coined the phrase “desirable difficulties” to describe the counter-intuitive notion that learning should be made harder by, for instance, spacing sessions further apart so that students have to make more effort to recall what they learnt last time. Psychologists at Princeton found that students remembered reading material better when it was printed in an ugly font.

Scientists from the University of Amsterdam recently carried out a series of experiments to investigate how obstacles affect our thought processes. In one experiment, people were set anagram puzzles to solve, while, as an obstacle to concentration, a series of random numbers were read out. Compared with those in a control group who performed the same task without this distraction, these subjects displayed greater cognitive agility: they were more likely to take leaps of association and make unusual connections. The researchers also found that when people are forced to cope with unexpected obstacles they react by increasing their “perceptual scope” – taking a mental step back to see the bigger picture. When you find your journey to work blocked by a construction site, you have to map the city in your mind.

As a poet, Ted Hughes had an acute sensitivity to the way in which constraints on self-expression, like the disciplines of metre and rhyme, spur creative thought. What applies to poets and musicians also applies to our daily lives. We tend to equate happiness with freedom, but, as the psychotherapist and writer Adam Phillips has observed, without obstacles to our desires it’s harder to know what we want, or where we’re heading. He tells the story of a patient, a first-time mother who complained that her young son was always clinging to her, wrapping himself around her legs wherever she went. She never had a moment to herself, she said, because her son was “always in the way”. When Phillips asked her where she would go if he wasn’t in the way, she replied cheerfully, “Oh, I wouldn’t know where I was!”

Take another common obstacle: lack of money. People often assume that more money will make them happier. But economists who study the relationship between money and happiness have consistently found that, above a certain income, the two do not reliably correlate. Despite the ease with which the rich can acquire almost anything they desire, they are just as likely to be unhappy as the middle classes. In this regard at least, F. Scott Fitzgerald was wrong.

Indeed, ease of acquisition is the problem. The novelist Edward St Aubyn has a narrator remark of the very rich that, “without the editorial influence of the word ‘afford’, their desires rambled on like unstoppable bores, relentless and whimsical at the same time.” When Boston College, a private research university, wanted a better feel for its potential donors, it asked the psychologist Robert Kenny to investigate the mindset of the super-rich. He surveyed 165 households, most of which had a net worth of $25m or more. He found that many of his subjects were befuddled by the infinite options their money presented them with. They found it hard to know what to want, creating a kind of existential bafflement. One of them put it like this: “You know, Bob, you can just buy so much stuff, and when you get to the point where you can just buy so much stuff, now what are you going to do?”

The internet makes information billionaires out of all of us, and the architects of our online experiences are catching on to the need to make things creatively difficult. Twitter’s prodigious success is rooted in the simple but profound insight that in a medium with infinite space for self-expression, the most interesting thing we can do is restrict ourselves to 140 characters. The music service This Is My Jam helps people navigate the tens of millions of tracks now available instantly via Spotify and iTunes. Users pick their favourite song of the week to share with others. They only get to choose one. The service was only launched this year, but by the end of September 650,000 jams had been chosen. Its co-founder Matt Ogle explains its raison d’être like this: “In an age of endless choice, we were missing a way to say: ‘This. This is the one you should listen to’.”

Today’s world offers more opportunity than ever to follow the advice of the Walker Brothers and make it easy on ourselves. Compared with a hundred years ago, our lives are less tightly bound by social mores and physical constraints. Technology has cut out much of life’s drudgery, and we have more freedoms than ever: we can wear what we like, sleep with whom we want (if they’ll sleep with us), and communicate with hundreds of friends at once at the click of a mouse. Obstacles are everywhere disappearing. Few of us wish to turn the clock back, but perhaps we need to remind ourselves how useful the right obstacles can be. Sometimes, the best route to fulfilment is the path of more resistance.