It was the fifth set of a semi-final at last year’s US Open. After four hours of epic tennis, Roger Federer needed one more point to see off his young challenger, Novak Djokovic. As Federer prepared to serve, the crowd roared in anticipation. At the other end, Djokovic nodded, as if in acceptance of his fate.

Federer served fast and deep to Djokovic’s right. Seconds later he found himself stranded, uncomprehending, in mid-court. Djokovic had returned his serve with a loose-limbed forehand of such lethal precision that Federer couldn’t get near it. The nonchalance of Djokovic's stroke thrilled the crowd. John McEnroe called it “one of the all-time great shots”.

Djokovic won the game, set, match and tournament. At his press conference, Federer was a study in quiet fury. It was tough, he said, to lose because of a “lucky shot”. Some players do that, he continued: “Down 5-2 in the third, they just start slapping shots …How can you play a shot like that on match point?”

Asked the same question, Djokovic smiled. “Yeah, I tend to do that on match points. It kinda works.”

Federer’s inability to win Grand Slams in the last two years hasn’t been due to physical decline so much as a new mental frailty that emerges at crucial moments. In the jargon of sport, he has been “choking”. This, say the experts, is caused by thinking too much. When a footballer misses a penalty or a golfer fluffs a putt, it is because they have become self-conscious. By thinking too hard, they lose the fluid physical grace required to succeed. Perhaps Federer was so upset because, deep down, he recognised that his opponent had tapped into a resource that he, an all-time great, is finding harder to reach: unthinking.

Unthinking is the ability to apply years of learning at the crucial moment by removing your thinking self from the equation. Its power is not confined to sport: actors and musicians know about it too, and are apt to say that their best work happens in a kind of trance. Thinking too much can kill not just physical performance but mental inspiration. Bob Dylan, wistfully recalling his youthful ability to write songs without even trying, described the making of “Like a Rolling Stone” as a “piece of vomit, 20 pages long”. It hasn’t stopped the song being voted the best of all time.

In less dramatic ways the same principle applies to all of us. A fundamental paradox of human psychology is that thinking can be bad for us. When we follow our own thoughts too closely, we can lose our bearings, as our inner chatter drowns out common sense. A study of shopping behaviour found that the less information people were given about a brand of jam, the better the choice they made. When offered details of ingredients, they got befuddled by their options and ended up choosing a jam they didn’t like.

If a rat is faced with a puzzle in which food is placed on its left 60% of the time and on the right 40% of the time, it will quickly deduce that the left side is more rewarding, and head there every time, thus achieving a 60% success rate. Young children adopt the same strategy. When Yale undergraduates play the game, they try to figure out some underlying pattern, and end up doing worse than the rat or the child. We really can be too clever for our own good.

By allowing ourselves to listen to our (better) instincts, we can tap into a kind of compressed wisdom. The psychologist Gerd Gigerenzer argues that much of our behaviour is based on deceptively sophisticated rules-of-thumb, or “heuristics”. A robot programmed to chase and catch a ball would need to compute a series of complex differential equations to track the ball’s trajectory. But baseball players do so by instinctively following simple rules: run in the right general direction, and adjust your speed to keep a constant angle between eye and ball.

To make good decisions in a complex world, Gigerenzer says, you have to be skilled at ignoring information. He found that a portfolio of stocks picked by people he interviewed in the street did better than those chosen by experts. The pedestrians were using the “recognition heuristic”: they picked companies they’d heard of, which was a better guide to future success than any analysis of price-earning ratios.

Researchers from Columbia Business School, New York, conducted an experiment in which people were asked to predict outcomes across a range of fields, from politics to the weather to the winner of “American Idol”. They found that those who placed high trust in their feelings made better predictions than those who didn’t. The result only applied, however, when the participants had some prior knowledge.

This last point is vital. Unthinking is not the same as ignorance; you can’t unthink if you haven’t already thought. Djokovic was able to pull off his wonder shot because he had played a thousand variations on it in previous matches and practice; Dylan’s lyrical outpourings drew on his immersion in folk songs, French poetry and American legends. The unconscious minds of great artists and sportsmen are like dense rainforests, which send up spores of inspiration.

The higher the stakes, the more overthinking is a problem. Ed Smith, a cricketer and author of “Luck”, uses the analogy of walking along a kerbstone: easy enough, but what if there was a hundred-foot drop to the street—every step would be a trial. In high-performance fields it’s the older and more successful performers who are most prone to choke, because expectation is piled upon them. An opera singer launching into an aria at La Scala cannot afford to think how her technique might be improved. When Federer plays a match point these days, he may feel as if he’s standing on the cliff edge of his reputation.

Professor Claude Steele, of Stanford, studies the effects of performance anxiety on academic tests. He set a group of students consisting of African-Americans and Caucasians a test, telling them it would measure intellectual ability. The African-Americans performed worse than the Caucasians. Steele then gave a separate group the same test, telling them it was just a preparatory drill. The gulf narrowed sharply. The “achievement gap” in US education has complex causes, but one may be that bright African-American students are more likely to feel they are representing their ethnic group, which leads them to overthink.

How do you learn to unthink? Dylan believes the creative impulse needs protecting from self-analysis: “As you get older, you get smarter, and that can hinder you…You’ve got to programme your brain not to think too much.” Flann O’Brien said we should be “calculatedly stupid” in order to write. The only reliable cure for overthinking seems to be enjoyment, something that both success and analysis can dull. Experienced athletes and artists often complain that they have lost touch with what made them love what they do in the first place. Thinking about it is a poor substitute.

We live in age of self-reflection, analysing every aspect of our work, micro-commentating on our own lives online, reading articles urging us to ponder what makes us happy. Much of this may be worthwhile, but we also need to put thinking in its place. Djokovic’s return was both the culmination of his life’s effort and an expression of careless joy. It kinda worked.