Candlestick Park, San Francisco, 1964. The wind is whipping off the Bay on a typically cold night at the ballpark. Mike Murphy takes his seat in Section 17. A jazz band pipes up and the vendors shout their wares: Hamm’s or Falstaff beers, Oscar Mayer hot dogs with Gulden’s mustard. Murphy is close enough to talk to the San Francisco Giants players – but he’s not interested in hero worship. He wants to put a voodoo curse on the opposition, the LA Dodgers.

He tells two friends it’s called a “whammy” or “occult backlash”. He’s been practising for years, perfecting the very particular cries and exact hand gestures to transmit negative energy to players. He reckons he’s a baseball witch doctor, sending psychic waves to scramble minds and zap energy from muscles.

While sitting in the bleachers at Candlestick Park, Murphy asked for assistance from the fellow Giants fans around him to explore his powers, explaining with a straight-face that the gestures had been developed by shamans in the Amazon basin to kill enemies. If they wanted the Giants to win, this would help. And so he exhorted the crowd to close their two middle fingers over the thumb, leaving the index finger and little finger pointing, like devil horns, towards their target. And he told them to shout and wail as they thrust their horns towards the Dodgers players.

That night would prove Murphy’s most successful as a conjuring cheerleader; according to his account, he enlisted almost 200 fans, all their negative energy flowing through him as he stood at the front, like the arrowhead. With several hundred horns pointing towards the tip, he began to feel dizzy. Whenever the wave of gestures and curses was at its strongest, the Dodgers began to make inept plays. The Giants went on to win.

Murphy staggered out of the stadium, drained, exhausted and fearing a heart attack. But believing that he had made it happen.

Baguio City, the Philippines, 14 years later. Mental combat has begun for the World Chess Championship. Anatoly Karpov, the golden boy of the Soviet Union, is playing Viktor Korchnoi, a defector the regime loves to hate. Despite sitting opposite each other for hour after hour, day after day, they have not spoken. But somebody is talking to Korchnoi. There is a voice inside his head. It is incessant. Over and over and over it berates him: “YOU. MUST. LOSE.”

Viktor Korchnoi and Anatoly Karpov compete for the 1978 World Chess Championship. Photograph: Jerry Cooke/Corbis via Getty Images

Korchnoi recognises the voice. It’s not his. It belongs to the man sitting in the front row of the audience since the match began. His heart starts to beat a little faster. He begins to sweat.

“YOU. SHOULD. STOP. FIGHT. AGAINST. KARPOV.”

The demands keep coming. Korchnoi is not afraid but he is angry. He understands perfectly what is happening. The man is trying to control his thoughts.

“YOU. ARE. TRAITOR. OF. SOVIET. PEOPLE.”

The man sits cross-legged, dressed immaculately in a white shirt and dark brown suit, reclining with a hint of arrogance. He looks like an accountant, albeit a somewhat demented one. A slight smirk plays across his face. His eyes are terrifying, bearing into Korchnoi. He does not blink until Korchnoi is defeated.

Both of these stories are true. Murphy, the zany hippy in bell-bottom jeans warbling occult orders, would, in time, have the US government dancing to his tune. And Dr Vladimir Zoukhar, the immaculately dressed communist spook, staring demonically for comrade and country, was considered the KGB’s mind control expert. Both men were protagonists in an extraordinarily paranoid chapter of human history: the cold war.

Murphy was no regular football fan. Known as “the godfather of the human potential movement”, he co-founded the Esalen Institute, a famed new age retreat and pillar of the counterculture movement in 60s California. It was a centre for eastern religions, philosophy, alternative medicines, and a fair amount of nude hot-tub bathing. Controversial eroticist Henry Miller swam at the hot springs in the grounds, Beatle George Harrison once landed his helicopter there to jam with Ravi Shankar, and Timothy Leary, whom Richard Nixon called “the most dangerous man in America”, taught regular workshops on the benefits of LSD, claiming that women could orgasm hundreds of times during sex when under the influence. And most recently, in the final frames of Mad Men, advertising executive Don Draper was seen smiling on Esalen’s lawn.

While Murphy was establishing Esalen, if Soviet state security wanted to place a negative or damaging thought in someone’s head, they called Zoukhar. That’s why Zoukhar was at Korchnoi’s match; communism trumped capitalism if it could produce a world chess champion. Korchnoi, hang-dogged and pot-bellied with his mistress in tow, was not the image they were going for. He could not be allowed to win against Karpov, the poster boy for true Soviet values.

‘He was Professor X, and Esalen was his Westchester Academy’ ... Mike Murphy. Photograph: Wally Skalij/LA Times via Getty Images

Murphy and Zoukhar hailed from opposite cultures teetering on the brink of nuclear Armageddon. But for all their differences, America and the Soviet Union held a common belief: the existence of superhumans. Both world powers believed in a race of cosmic beings who could, just like in the sci-fi movies, slow down time, speed it up, change their body shape, feel no pain, levitate, see into the future, and more. With boggle-eyed mind control and harnessing the occult, both nations believed they could put a thought in someone’s head, or stop a man’s heart at 100 paces. Both nations thought these powers would win them the war. From the west coast of America to the far corners of the Soviet Union, yogis, shamans and psychics were sought out to aid these alternative war efforts, with millions spent on attempts to create a real life Superman or Wonder Woman.

Since the first millennium, Hindus and Buddhists have believed that spiritual practice – like yoga – was capable of giving rise to a siddhi, a Sanskrit word that roughly translates as “superpower”. Siddhis included an ability to master pain, levitation, invisibility, being able to read minds at a distance – basically, any comic book superpower you can think of. And in America, Murphy was Professor X, and Esalen was his Westchester Academy.

As a trust-fund kid at Stanford University, Murphy had once wanted to be a priest, maybe a scientist; his parents preferred doctor. But on the second day of spring classes in 1950, Murphy went to the wrong classroom and ended up listening to a comparative religion lecture. It was providence. Murphy was hooked. He quit his class, enrolled in Indian philosophy and devoured The Life Divine, a 1939 book by Indian mystic, yogi, guru and poet Sri Aurobindo. It was a handbook for spiritual powers. From then on, Murphy would dedicate his life to the pursuit of the extraordinary.

Hunter S Thompson was Esalen's first security guard; Aldous Huxley, then often high on mescaline, helped establish it

It just so happened that there was a big chunk of Murphy family land on the cliffs of Big Sur, California, which would do nicely as a base for teaching superpowers. Murphy provided the land , and his friend Dick Price, a co-founder, brought the cash. Before Esalen even opened, in 1962, the plot needed a security guard; a young Hunter S Thompson, pioneer of gonzo journalism, got the job.

“He was 21,” Murphy told me, on our first phone call. “Unpublished. Fully armed. With a small arsenal. He seemed to love tracer bullets. He’d fire hundreds out into the night sky so every night was like the fourth of July. Sometimes he’d fire his gun out of the unopened window of one of the houses that was on the grounds.”

Thompson was the first of many colourful and culturally significant characters in the Esalen story. Aldous Huxley, the English novelist and psychedelic grand philosopher, followed. He was another inspiration for Murphy, who attended a Huxley lecture at the University of California with Price, entitled Human Potentialities, in 1962. Huxley, then often high on mescaline, was a key figure in Esalen’s establishment, before his death a year later from throat cancer.

Murphy believed that the best place for the superpowers to reveal themselves was on the sports field. Sports, he said, was the west’s yoga. He collated thousands of stories of athletes describing siddhis experiences. John Brodie, the San Francisco 49ers quarterback, once told Murphy that he could move the ball with the power of his mind and reported seeing rivals shape-shift to avoid tackles. Morihei Ueshiba, the founder of the martial art aikido, was said to be able to make himself invisible. Murphy found “scores” of reliable witnesses who had seen Ueshiba elude attacks, with one writing: “Completely surrounded by men with knives, Ueshiba disappeared and reappeared at the same instant, looking down at his attackers from the top of a flight of stairs.”

What about a human passing through solid matter? Murphy cites Pelé, one of the world’s greatest footballers, as once saying: “I felt that I could dribble through any of their team … that I could pass through them physically.” In 1970, ice hockey star Bobby Orr was said to have called on abilities of thought projection to hypnotise Chicago Black Hawk players during a four-game winning sequence on the way to the Stanley Cup finals for his Bruins team.

Bobby Orr leaps with joy after scoring the goal that won the 1970 Stanley Cup for the Boston Bruins. Photograph: Ray Lussier/AP

Unsurprisingly, the American military and CIA began to take note of Esalen. Prompted by intelligence reports about the Soviets experiments, the Americans felt they had to start programmes of their own, in which Murphy would become a key figure. His friendship with Democratic senator Claiborne Pell, a supporter of Esalen and Murphy’s jogging partner, was instrumental. In White House intelligence meetings, Pell spoke forcefully of the benefits of supporting their experiments. He argued that if the Russians had it, and the Americans didn’t, they would be in serious trouble.

Murphy was an adviser for the Jedi warrior training programme at West Point Military Academy in New York. Code-named Project Jedi, soldiers in the programme were taught invisibility, seeing into the future and extraordinary intuition, like knowing how many chairs were in a room before walking in – but also stopping the hearts of animals. It was similar to the First Earth Battalion (FEB), best known from the movie The Men Who Stare at Goats. Jim Channon, the founder of the FEB, thought it a good idea that each of his “warrior monks” should carry pouches of herbs into battle, give out flowers as a sign of peace, and play indigenous music to calm and confuse the enemy. Channon had been given a small Pentagon budget and two years to research ways for the US military to use new age methods in warfare. He spent most of that time at Esalen, being taught by Murphy.

In 1975, the Chicago Tribune reported that the CIA was attempting to develop a new kind of “spook”, after finding a man who could “see” what was going on anywhere in the world. CIA scientists would show the man a picture of a place, and he would then describe any activity going on there at that time.

In fact, there was more than one of these men. Russell Targ, who had taught this psychic power at Esalen, was one; another was Uri Geller. (You might have heard of him and his bendy spoons.) There was a whole team of psychics based at the Stanford Research Institute (SRI) in Menlo Park, California, as part of the CIA’s Stargate programme to find psychic warriors. Targ and Geller would sit in that office, close their eyes, breathe deeply and then after a few minutes draw the location of Soviet missiles. Sometimes, they were right.

By contrast to the Soviet plan, Targ and Geller seemed harmless. “They were using it to kill people,” Targ said. The Russian term for superpowers was “Hidden Human Reserves”.

Senator Claiborne Pell, a high-profile supporter of Esalen, pictured in 1972. Photograph: Bettmann/Bettmann Archive

“They created a replica of the Oval Office and they would have people in there 24 hours a day concentrating on the US president in such a way to try to muddle his thinking,” said Jim Hickman, a key figure at Esalen during the cold war. Hickman spent much of the 70s and 80s travelling in the USSR to research siddhis, often with Murphy at his side. Hickman told me they learned about strange, disturbing things. Even stranger and more disturbing than a whole bunch of Dr Zoukhars trying to influence the thought patterns of a head of state from great distances? “There was much deeper work going on,” he said. “We knew that we were only talking to the people they let us talk to.”

In one creepy experiment, scientists implanted electrodes in a mother rabbit’s brain, took her young litter off to a submarine and killed them one by one

Those people were probably among the 60 Soviet-based scientists who worked at what was called Special Department 8. Their job was to investigate distant mind control. It was just one of 40 centres at Science City, Novosibirsk, in south-west Siberia, which housed thousands of scientists and their families from across the communist bloc, in a kind of nerdy utopia. Road names included Calculators Street, Thermophysics Street and Hydrodynamics Square. Here, Soviet scientists were attempting to prove the existence of extrasensory perception (ESP). In one creepy experiment, the scientists implanted electrodes in a mother rabbit’s brain, took her young litter off to a submarine and, when it was deep below the surface, killed them one by one. At each synchronised time of death, the mother’s brain reacted.

At the Kharkov University Neurology Institute, the rat brains were attached to electrodes and put in solution. The best Russian psychics, having been tested in research centres dotted around the state, were brought in to transmit emotions and thoughts to the brains. The most popular response recorded was laughing but the brains also “enjoyed” sums. (It is not known whether the brains were better at fractions or algebra.)

Unsurprisingly, some of the morally questionable experiments began to make the experts feel uncomfortable. One lab was even shut down in 1974 as the resident scientists en masse rejected what they called “the negative work”.

It is widely believed that a mega-secret lab was set up in its place, in a sub-sub-basement below the Filatov Institute in Odessa. Only clandestine couriers knew how to access these secret paranormal departments. KGB guards made sure there were no unwanted visitors. There, death row prisoners were “bombarded” with pulsing magnetic fields to see if they would become clairvoyant. Years later, in 1991, one Dr Bryukhanov would publicly claim that he had run the project and said they had believed that animals they tested it on had developed the ability to see through walls. Alas, their small animal brains could not cope with the onslaught from the magnetic fields, and simply disintegrated. The prisoners reportedly suffered the same, horrible fate.

A nude encounter group therapy session at the Esalen Institute, in 1968. Photograph: Ralph Crane/The LIFE Picture Collection/Getty Images

By 1984, the cold war was heating up. The Doomsday Clock, the timepiece those cheery folk at the Bulletin of Atomic Scientists kept ticking to judge how close man-made global catastrophe was, nudged the little hand to three minutes to midnight and oblivion. Luckily, help was at hand in the most unlikely of places. The hippies, gurus, shamen and yogis wanted to start a thaw and they knew exactly how to do it: nude hot-tub bathing.

With their friends in Washington giving them a nod and a wink, a series of Soviet – American exchanges had been organised from 1980. After several visits, Murphy and Hickman revealed they had discovered a Soviet Esalen: a group of believers and free thinkers who believed in the existence of siddhis. Contacts ranged from the obligatory psychic to Kremlin influencers. As a result, astronauts and cosmonauts, writers, KGB agents, military veterans, politicians (like Claiborne Pell) and diplomats were frequently invited to Esalen. There, they were told to sit crosslegged on the floor with pillows and just talk to each other. Then it was time to jump in the hot tub.

The idea behind the exchanges was to convince Americans and Russians to recognise that they were not so different after all. The brains at Esalen wanted to call this project The Institute for Theoretical Studies. When someone pointed out that the acronym for that was TITS, it was renamed the Esalen Soviet-American Exchange Program. It still exists today, albeit under a different name – Track Two: An Institute for Citizen Diplomacy – and is run by Murphy’s wife, Dulce.

“We had KGB guys and CIA guys at Esalen talking to each other,” Murphy said. “The KGB knew communism wasn’t working. It was their job to know everything … they could fucking see it. When we hosted people, it was just confirming what they already knew.”

The exchange programme’s biggest coup was bringing Boris Yeltsin to America in 1989. At the time, as a critic of the communist regime, Yeltsin was considered a political lightning rod. It was a visit that played a part in ending the cold war.

Yeltsin’s aides had contacted Russian activists connected to Hickman and Murphy, and took part in Esalen’s exchange programme.They asked the institute to ask if they would be interested in hosting Yeltsin.

“It turned out to be a gigantic deal ’cos he flipped!” Murphy said.

When Boris Yeltsin visited a Houston grocery store, he sobered up quickly. He was shocked by the bountiful aisles – with no queues – and felt sick with despair

Yeltsin spent most of the trip drunk. But when he visited a Houston grocery store, called Randall’s, he sobered up pretty quickly. One of the Communist party’s great lies was that America staged its wealthy image through fake stores. He was shocked by the bountiful aisles of meats, cheeses and vegetables. He asked why no one was queuing. He stopped shoppers to ask how much they earned per month and what they spent on food. Yeltsin became upset. “When I saw those shelves crammed with hundreds, thousands of cans, cartons and goods of every possible sort, for the first time I felt quite frankly sick with despair for the Soviet people,” Yeltsin later wrote in his autobiography. Two years after that supermarket trip, he quit the party, stood on a tank in Red Square and became a capitalist. It was the start of the Iron Curtain’s fall.

But neither side stopped searching for a superman. Murphy, now approaching his 90s, is still looking. Every year he holds a conference, the Sports, Energy and Consciousness festival in San Francisco, where he holds workshops advising people how they can experience the siddhis. The American military haven’t stopped, either, spending millions of dollars a year on research for “alternative powers”. And just like the Russians and their chess experiment, they are using the sports field as a training ground.

• This is an edited extract from Ed Hawkins’ new book The Men on Magic Carpets: Searching for the Superhuman Sports Star (Bloomsbury, £16.99).