I was in San Francisco last week, visiting my brother and revisiting the years we spent there as young men. We walked through Golden Gate park, two guys in their sixties, admiring the giant sequoias, exotic gardens, and gold-green pastures cascading westward to the ocean. And we reminisced about our epic acid trip back in 1969, in that very place, when we were primed for adventure and self-discovery.

I wouldn't encourage anyone to soothe existential discomfort with heroin or amphetamine. But psychedelics have a value

There were four of us. We each swallowed our little purple tablets – “Happy landing, guys!” – then started walking westward through the park. About a half-hour later came surges of energy in our stomachs, anticipation of something huge about to happen, excitement and fear as though approaching the brink of an unknown waterfall. The park was so beautiful. Bird song was everywhere, more nuanced than I’d ever noticed.

And then the world transformed. The patterns of branches, the smell of the eucalyptus, the hubbub of other people, barking dogs, twittering birds, blended and disintegrated at the same time. My sense of “I” stretched, extending the boundaries of my perception like an unfurling bouquet. I lost track of who I was, where I was, where I stopped and the outside word began … and then all that disintegrating, reconverging and reshuffling somehow collapsed into a sonorous oneness, the deafening drone of an utterly peaceful universe.

LSD was frightening, beautiful, and incomprehensible, life-changing – you might say “educational” with a wry grin – for the uninitiated. Not only that. Along with other psychedelic drugs it fashioned our subculture, became symbolic of a unique time and place. It was the beacon for a bunch of kids intent on discovering something true and universal underlying the political and social absurdities of mainstream society, Nixon, the Vietnam war, and all the rest of it.

What we sought in LSD is what humans have always sought – meaning hidden behind the transitory stupidity of human strivings that lead nowhere. What we sought was the enlightenment that the Buddha yearned to show us, and that for thousands of years has remained so elusive.

Was this just idealistic folly, drug fever? Did we have our heads up our butts, imagining ultimate beauty because we were so completely in the dark, so expertly narcissistic, so uninspired by the challenges that our parents and figureheads wished us to face?

Facebook Twitter Pinterest A group of drummers entertains protesters during the People’s Park demonstrations in 1969, which the author attended during his student days in California. Photograph: Ted Streshinsky/Corbis

‘The Higgs boson of neuroscience’

When I got home again, three days ago, a couple of neuroscientific experiments had just been published in two journals simultaneously, claiming to show exactly what went on in the brain when people took LSD. Within two days, these findings migrated from scientific journals to newspapers and media channels around the world, including the Guardian. One of the work’s senior authors, David Nutt, a respected neuropharmacologist, was quoted as saying “This is to neuroscience what the Higgs boson was to particle physics.”

Though that might overstate his case, we can forgive Nutt his enthusiasm: the findings are indeed extraordinary. Healthy volunteers were injected with LSD while lying in an MRI scanner, and subjected to several other neuroimaging methods at the same time. This amounted to an arsenal of measurement that previous decades of psychedelic researchers could only dream of. It wasn’t Golden Gate park to be sure, but lying in that magnetic chamber, with their eyes closed and their brains open, with streams of numbers spilling from their lobes into data files soon to be translated into images, these individuals watched LSD’s intricate hallucinations unfold behind their eyelids. And at the same time they reported experiences of “ego dissolution” very much like those we experienced in the park.

The normal etiquette of the brain requires segregating parts with different functions. That etiquette was blown to bits

What was most remarkable about the research is that the degree of ego dissolution reported by the participants correlated with a specific neural transformation. To get through the pragmatics of day-to-day life and the demands of survival, brain activity naturally differentiates itself into several distinct networks, each responsible for a particular cognitive function.

The three networks most closely examined by these scientists include a network for paying attention to what’s most salient, a network for problem-solving, and a network for reflecting on one’s own past and future. There is also a natural segregation between high-level (abstract) cognitive areas and low-level (concrete) perceptual areas, most notably the visual cortex. These distinctions are thought to be an essential design feature of a functional human brain.

The impact of LSD was to diminish connections within each of these networks, relaxing the bonds that kept them intact and distinct, while increasing the cross-talk among them. In other words, the normal etiquette of the brain requires segregation among networks that have different functions, and that etiquette was blown to bits.

Now most parts of the brain were communicating with most other parts of the brain. Concrete sensory experiences, like vision, intermingled with cognitive abstraction, and cognitive abstractions reshaped visual imagery. Perhaps that’s what explains the intricate fractal elaboration that people see in the branches of a bush while tripping on acid. The perception of salience and refinement of a sense of self are hashed together like potatoes and gravy. The brains and their owners no longer distinguish between what is most important, how to get stuff done, and who in fact is the arbiter of the importance of the stuff that needs to be done.

Facebook Twitter Pinterest The brains of subjects lying awake with their eyes closed, under a placebo, left, and the drug LSD, right, visualized using functional MRI. Photograph: Imperial College London-The Beckley Foundation/Reuters

Religion v psychedelics

Some thousands of years ago, the Buddha defined human personality as the recursive cycling of habits – habits of acquisition, of craving and grasping. They led to the pursuit pleasures that were sure to fade and the avoidance of suffering that could not be avoided in a cycle of life, ageing, and death. In response, his followers chose asceticism, the practice of over-control. From the Buddhist monks who stripped themselves of comfort and the Hindu sadhus with their rituals of self-mortification evolved new religions.

There came endless lists of edicts that Jews, Muslims, and Christians still impose on themselves: what one was not allowed to do, on which days, with what consequences if one failed. Our attempts to wrest freedom from habit, universality from local custom, truth from delusion, have generally amounted to a set of rules to increase control, to partition and segregate, to fashion hierarchies and obey codes.

It seems that our brains, with their intrinsic tendency to parse and segregate, were well designed to veer toward over-control in response to the hardships of existence. Or, more accurately, we came by our tendency toward over-control because it manifests a key principle of brain design.

But nature provided us with a different antidote to isolation and irrelevance. LSD was created in a lab in Switzerland in the 1930s. But other chemicals with the same psychedelic properties dwell in the flesh of cactuses throughout North America (mescaline), mushrooms found across much of the northern hemisphere (psilocybin), and the vines of the Amazon (DMT–ayahuasca). These naturally evolved chemicals undo the locks our brains construct to keep us on the straight and narrow, pursuing short-lived victories over inevitable failures.

Humans have gathered, cultivated, distilled, and manufactured all kinds of drugs for thousands of years. Some of them relieve pain and bestow comfort. Others provide the energy we sometimes need to complete our tasks. And our old friend alcohol helps us relax and have fun. But the psychedelics contribute nothing to our day-to-day functioning. Rather, we use them to see the bigger picture, to connect with a reality that is difficult to see using our normally functioning brains. We are literally small-minded most of the time. And though meditation and mindfulness nudge us toward openness, acceptance, and relinquishment of our egos, humans continue to turn to psychedelics to wake us up to the possibilities of a universal perspective.

Not all drugs are created equal, and I would never encourage anyone to soothe their existential discomfort with heroin or amphetamine, both of which I have taken. But psychedelics have a value I can’t help but admire. And now we understand more about how they do what they do. A simple code unlocks the gates in our brains, gates that normally act as walls.

I hope that these discoveries dispel just enough mystery to encourage us to keep exploring.