A few years ago, beset by the same malaise that I suppose afflicts everyone who spends too much time in the bustle and chaos of a big city, I wondered if solitude might be the answer. I began to read about hermits and became obsessed with the idea of meeting one.

As you might imagine, hermits are a difficult sub-group to track down. But I found out about a newsletter run by a couple in the Carolinas aimed at solitaries and, after posting an ad there, began writing to a few.

The correspondences never led anywhere. The closest I got to an actual encounter was with a woman in rural Oregon called Maryann. We planned to meet but at the last minute she got cold feet, writing to say she could not risk letting a stranger visit her “in this crazy age of violence”.

It was winter by then. Desperate to flee the city, I flew to Vegas with a vague plan to hitchhike in to the high deserts of Arizona and New Mexico, which I had heard were good hermit hunting grounds.

In the canyons of central Arizona, in Cleator, an inglorious little town of tin-roofed cabins an hour’s meandering drive west of the interstate, I heard about a man who had lived alone for 20 years guarding a disused silver mine. The next day I walked up the mountain to find him, watching the ground for rattlesnakes as I went.

I had high hopes; I had read accounts of those who had gone alone into the wild and come back laden with deep personal insights. I wasn’t exactly expecting the Buddha, but a minor-league Thoreau would have been nice.

As it was, I met Virgil Snyder. The first thing he asked was if I had brought beers. I had, and for the rest of the day I watched him down them, one after the other at his cabin, a ramshackle place cluttered with old birds’ nests and the bleached skulls of pack rats he had found on the trail.

Virgil’s home in central Arizona. Photograph: Paul Willis

He didn’t understand why I had come. When I told him I was interested in learning about solitude, he looked at me like I had just flown in from Planet Stupid.

“I didn’t come here to prove a point,” he said. “I don’t do this to be unique.”

I wrote down everything he said, poring over my notes at night, searching for some searing insight among his professed hatred of, well, everything, and the litany of insults he had thrown my way. (I was at different times called “a faggot”, “a motherfucker” and, more bizarrely, “a Tootsie Roll”.)

After several visits, I was forced to admit that he was not the mountain sage I had been looking for. He was an angry drunk.

• • •

The idea that those who withdraw from the world accrue great wisdom is an old and powerful one. In Hindu philosophy, all humans ideally mature into hermits. As the Indian guru Sri Ramakrishna put it: “The last part of life’s road has to be walked in single file.”

In the west, the idea has had a profound cultural impact. Peter France explores this in his book Hermits, attributing the creation of monasticism to the example set by the earliest Christian hermits, the Desert Fathers of Egypt.

One of the historical ironies France notes is the way hermits have been sought out for their advice on how to live in society. The Desert Fathers’ thoughts were considered so valuable that a collection of their sayings – known as the Apophthegmata – were written down in the late fourth century. In Russia, 19th-century hermit Startsy Ambrose’s fame drew illustrious visitors like Dostoyevsky, who consulted the hermit several times following the death of his son; their encounters were immortalised in The Brothers Karamazov.

The trend continues today, most notably in the case of the so-called North Pond Hermit. A Maine native, Christopher Knight lived alone in the woods without human contact for 27 years; his story came to light only after he was arrested for a spate of robberies in 2013.

Michael Finkel, the author of the GQ article that brought Knight to wider prominence, was similarly obsessed by the idea that the hermit had some “grand insight” to share from his time in the wilderness. In the piece – reportedly the most-read GQ article ever – Finkel keeps pushing Knight on the subject and at one point it seems like he is about to spill the beans.

“I felt like some great mystic was about to reveal the Meaning of Life,” Finkel writes. Except all Knight has to offer is, “Get enough sleep.”

• • •

The same afternoon that I left Virgil, a Catholic monk I had been corresponding with left a message on my phone to tell me about Doug Monroe, a religious solitary who had been living alone for a decade in New Mexico’s vast Gila Wilderness.

The monk described Doug as an “exceptional soul” and his hermitage as “the real thing”. There was no road or habitation within 10 miles of him and apart from a trip to Albuquerque once a year to restock his supplies, the monk said that he never left the cabin.

Buoyed by the serendipity of the timing I decided to go find him. The route to Doug’s place switched back and forth across a stream gushing with snowmelt.

Doug at home. Photograph: Paul Willis

I was greeted like a long-lost friend. “Boy, it’s such a treat to have ya here,” Doug said in a homely southern accent, fussing over me, feeding me rice and tea.

Unlike Virgil, he understood my interest and tried to convey what the solitary life was like. He described moments when the silence around him was so profound it left him frozen to the spot, afraid that the noise of even one footstep would be deafening.

The desire to be a hermit had first come to him in his mid-20s, he said, but it was not until his late 40s that he finally plucked up the courage. When he first came here he had just $150 in cash and an 80lb pack on his back and trekked out into the forest determined to “entrust my survival to God”. For the first year, he lived in a metre-wide shelter he built below an exposed rock face using slabs of stone and fallen trees.

He eventually built himself a one-room cabin. Compared with the melancholic decay of Virgil’s home, there was a calm order here: all his supplies were arranged neatly around the room. On the shelves were boxes of crackers, bucket-sized tubs of peanut butter, dried milk and grains, tins of tuna and Spam, cocoa and powdered mash.

On the wall were photos of the family of his benefactor – a businessman and devout Catholic – in Albuquerque. On Doug’s annual excursion in town, the benefactor takes him to a wholesaler and buys him yearly supplies with change from $1,000.

Next, Doug took me outside to show me the 6ft-deep well he had built in a small creek. Piping ran from the well to the water tank that sat on raised ground behind the cabin and he had a small generator to power the pump.

As I followed him around, I thought about how Doug’s experience with solitude was nothing like Virgil’s. While Doug’s faith gave his life in the wilderness a structure and a purpose, that was completely absent with Virgil.

Apart from a rudimentary contraption for trapping rainwater, I had seen few clues about how Virgil survived in Arizona. He had hinted at well-wishers bringing him supplies, though when I pried further he refused to be drawn. Perhaps it would have undermined his hermit status, which I think he secretly enjoyed, despite claiming he didn’t care what folk called him.

Doug’s one-room cabin. Photograph: Paul Willis

I had the sense that Doug was genuinely content with the path he had chosen, but there was an eccentricity I saw in him too. He talked non-stop, jumping from one subject to the next without any clear connection. At first I thought he was just excited by my presence but he admitted that it was the same when he was alone. He held imaginary conversations with absent friends, with dead saints, even with the Virgin Mary.

He said his inability to stop talking went back to childhood – he estimated he could have filled an encyclopaedia with all the lines he wrote for talking in class – but it crossed my mind that the solitude might be exaggerating the trait.

Solitude, after all, is known to do strange things to the mind.

• • •

In 1993 the sociologist and caver Maurizio Montalbini broke the record for the longest time spent underground, during a spell in a cavern near Pesaro, Italy. During his isolation, Montalbini began experiencing a slowing down of time. His sleep-wake cycles nearly doubled in length so that when he finally emerged he was convinced only 219 days had passed whereas in fact a year had elapsed.

While there are numerous studies showing the harmful effects of solitary confinement on prisoners, studies of the general public are rarer because of the ethical concerns around subjecting someone to prolonged isolation for the purpose of a clinical trial.

Back in the 1950s, however, Donald O Hebb, a professor of psychology at Montreal’s McGill University, did just this. Hebb had his volunteers spend days, or even weeks, in sound-proof cubicles, deprived of human contact.

After a few hours, the subjects became restless, talking to themselves to break the monotony. Later they grew anxious, highly emotional and their cognitive abilities began to wane as they struggled to complete arithmetic and word association tests. At some point many began having hallucinations, both visual and auditory. One man even hallucinated being shot in the arm and felt the sensation of pain. The subjects became so disturbed that the trial was cut short.

The most notorious example of the mind-distorting effects of solitude is the case of Donald Crowhurst, who took part in a 1968 race to become the first solo sailor to go non-stop around the world. From the race’s outset, Crowhurst ran into problems with his boat and, faced with the prospect of returning home a failure, he sailed aimlessly around the Atlantic while sending back false reports of his position.

Fearing financial ruin and overwhelmed by the scale of the subterfuge, he cut radio contact. His boat was discovered floating in the Sargasso Sea months later. Crowhurst was nowhere to be found, but a 25,000-word diary discovered on board detailed the Englishman’s descent into madness.

During one visit to Virgil, I found the door to his cabin open and Virgil passed out at the table, an empty liquor bottle beside him. Afraid of his reaction if he suddenly came to and found me there, I went outside and knocked hard till he stirred. When he finally emerged he stared at me like I was a ghost.

On Virgil’s property grounds. Photograph: Paul Willis

It was a tense encounter, his mood volatile. One minute he erupted in anger, upsetting beer cans and thrusting a finger in my face, and the next he was crying uncontrollably. At one point he blurted out about a wife and two kids he had been estranged from for nearly 30 years. When his marriage broke down he lived destitute on the streets in Phoenix, he said. His father, who was caretaking another silver mine further down the mountain at the time, found him and brought him back in his pickup. After a few years the old man drank himself to death.

“Big fucking deal!” he said at the story’s close. “What do you care!”

• • •

Among the Apophthegmata is a saying by an unknown hermit: “It is better to live among the crowd and keep a solitary life in your spirit than to live alone with your heart in the crowd.”

In other words, if you go into solitude to get away from something, your troubles will probably follow you. This, I suspect, was Virgil’s story. It was probably my own, too, and I returned to the city unhappy that my hermit encounters had not yielded more. To my disappointment, Virgil and Doug had proved all too human.

There was one aspect of the experience that had surpassed my inflated expectations: the environment where the two men lived. And as I became entrenched once again in city life, it was to the stark beauty of the high desert in winter that my mind kept returning, to the saguaros, dwarf junipers, pinyon pines and magical starlit nights.

In the 1968 race that cost Donald Crowhurst his sanity, another competitor had a very different experience.

French sailor Bernard Moitessier fell utterly in love with life alone at sea. So much so that instead of turning north towards the finishing line in England and possible victory, he dropped out of the race and sailed on to Tahiti.

In his book The Long Way, Moitessier describes sailing one night by a headland with the Milky Way overhead. It occurs to him that were this view only visible once a century, the headland would be thronged with people. But since it can be seen many times a year the inhabitants overlook it.

“And because they could see it almost any night, perhaps they never will,” he writes.

It was a direct encounter with the quiet magnificence of nature that was the real gold I brought back from my wanderings in Arizona and New Mexico. It was probably what I had been looking for all along.