Jaron Lanier has written a book about virtual reality, a phrase he coined and a concept he did much to invent. It has the heady title Dawn of the New Everything. But it’s also a tale of his growing up and when you read it, what you really want to talk to him about is parenting. Lanier is 57, but his childhood as he describes it was so sad and so creative and so extreme, it makes him almost seem fated to pursue alternative worlds.

Lanier’s parents met in New York. His mother, Lilly, blond and light-skinned and Jewish, had somehow talked her way out of a “pop-up concentration camp” in Vienna after the Anschluss, aged 15. The family of his father, Ellery, had escaped a murderous pogrom in Ukraine. They met as part of a circle of artists in Greenwich Village in the 1950s. Lilly was a painter and a dancer, Ellery an architect, but when Jaron was born in 1960 they moved to El Paso, Texas, right on the border with Mexico. Lanier was never sure why, but he believes it was an effort, given their own childhoods, to “live as obscurely as possible”, off grid. His mother did not trust American schooling, so he went across the border to a Montessori school in Mexico each day; then, after a change of heart, to a Texas public high school, where he was bullied.

When Lanier was nearly 10, his mother was killed and his father severely injured in a car crash. The accident happened after his mother had seen Lanier assaulted by bullies on the way to school. He feared the two events were connected, that she had been anxious or distracted; much later he learned the car she was driving most likely had a fatal fault. After his mother’s death he fell ill with a succession of infections, including scarlet fever and pneumonia, which kept him in hospital for a year.

Of his time living in the desert, he says 'It felt as if we were living in the frame and not the painting of the world'

During this time their house in El Paso burned down and, unemployed and grieving and virtually penniless, his father bought a parcel of uninhabited land in the New Mexican desert for them to live on. Ellery allowed his son to design their new house, which he based on the geodesic domes of R Buckminster Fuller, all the rage with hippies. This was 1972. The dome took two years to construct, and in the meantime father and son lived in an army surplus tent, bone cold in winter, deep fried in summer. They never talked about his mother. Lanier still hated school, but developed a passion for music, and for technology.

Their closest neighbours worked at the White Sands Missile Range, out in the desert. One was the astronomer Clyde Tombaugh, who had discovered Pluto as a young man and who taught Lanier to grind lenses, and let him play around with the homemade telescopes in his back yard. Almost unable to communicate with strangers, but with a precocious talent for maths, Lanier took classes at New Mexico State University aged 15 or 16, and then at 17 transferred to Bard college in New York, paying for tuition by selling goat cheese from a herd of goats he had bred.

Returning to New Mexico, he fell in love and followed his girlfriend (whom he had serenaded on their first date, in a laundrette, with a Japanese bamboo flute) to California, where she finished with him. He found himself alone in the start-up land of Silicon Valley, with a head full of equations that didn’t all add up and a hankering for different worlds. At this point in his memoir, not surprisingly, Lanier turns outward to the reader: “You might be thinking by now that this book is a work of magical realism,” he suggests.

Talking to him on the phone last week, I admit that I did have my doubts. He hoots, with his high laugh. “At the time,” he says of the desert years, “it felt almost as if we were living in the frame and not the painting of the world. My personal experience was so different from anything else, I couldn’t even compare, really.”

Lanier now lives in Berkeley, California, with his wife, Lena, a child psychologist, their 11-year-old daughter, Lillibell, and more than a thousand musical instruments, ancient and modern, all of which he tries to play. Having made and lost a paper fortune with his pioneering virtual reality headsets, he sold an interest in another company involved in face recognition software to Google in 2006. Since then he has had an innovation lab at Microsoft, and become a prominent critic of the ways in which technology and social media have shaped our world (a criticism “driven more by fear than love”), most notably in his bestselling book You Are Not a Gadget.

One aspect of that latter argument is a belief that we should never separate a discussion of technological advance from its human effect. In his previous writing, Lanier says, he has sometimes adopted a theoretical or abstract tone, as if issues around virtual worlds and artificial intelligence had an independent life of their own. He has used the autobiographical voice in the current book partly as a statement of intent. It wasn’t easy for him.

“I kind of coped with my mother’s death and lots of other things by putting them out of [my] mind,” he says. “Having to encounter that again was difficult. But I am unhappy with the way that digital technology is influencing the world, and I think the solution is to double down on being human…”, which leaves Lanier no choice but to put himself all the way into his book.

Perhaps one of the effects of that is to root his advances in virtual reality in very human psychology. At one point in the book, he recalls how he and his geeky friends used to fantasise about putting a “4D headset” on a baby immediately after birth and imagine the strangeness of the world it would experience. I wonder if his own parents had something of the same experimental impulse with the simpler tools they had to hand. He giggles again, but says that was probably down to the traumas of their own childhoods. “I think that people who have suffered horrible things do have some extra – almost magical – investment in their children,” he says. Even so, he is not sure that his father allowing him to design their desert home was the greatest idea. He admits he is a little nervous that someone will read his book and see it as a challenge: “I do want to remind people that the dome did subsequently collapse.”

He also says that when he arrived in Silicon Valley he found like-minded twentysomethings among tech entrepreneurs and hackers: the children of commune dwellers and peace protesters, who had been brought up with the liberal regime of the childcare guru Dr Spock and who recognised no limits to imagination, and often ego. Counterculture fed directly into plutocratic tech culture.

Lanier with proto-VR kit in 1990. Photograph: Rex Features

Lanier is often asked whether his interest in musical instruments came from the same place as his interest in virtual worlds. He has no doubt that it does. “I still just get a tremendous joy from learning new ones,” he says. “I have been working just now with an Ethiopian instrument called a begena, an old harp. Probably similar to the one David played in biblical times. The way you have to hold it is interesting. That kind of thing enthrals me. It is like time travel… it brings your body’s movement into some kind of a connection with people who lived many centuries ago.”

In virtual reality terms, Lanier would perhaps call this connection “haptic”, a way of being in intimate touch with fragmenting external realities. One thread that seems to connect all of his preoccupations over the years is a restless effort to find new arenas in which to communicate with other people, as if the conventional ones are not enough. Bluntly, I ask whether he traces this impulse back to the trauma of losing his mother, and his subsequent isolation.

“Yes, obviously so,” he says. “But then immediately after that I would say: ‘Does a desire to communicate make me different from other people, or does it expose a commonality?’ I think it is more the latter. I think we all want something deeper.”

In the early 1990s Lanier appeared to believe he might find that depth in the computer-simulated environments he tried to create. He became friends with the likes of Timothy Leary, the philosopher of hallucinogens, and though he never took any drugs himself, Lanier was standing by in case of accident when the Nobel prize-winning physicist Richard P Feynman first experimented with LSD in a hot tub in Big Sur, California.

Lanier believed that virtual realities could have an impact on the doors of human perception comparable to that promoted by the early disciples of psychedelia. He was not alone in this curiosity. On one memorable afternoon he gave demos with his first headsets to Terry Gilliam, the Dalai Lama and Leonard Bernstein. The technology was not without its constraints, however. One early prototype set of goggles had to be ballasted with sandbags to allow the viewer to remain upright.

He believed that virtual realities could have an impact on the doors of human perception comparable to early psychedelia

Lanier hasn’t lost all of that faith, but he despairs of how the utopian vision of that early hacker culture was so quickly corporatised. He says he was always alive to those dangers.

“In 1995 I wrote this essay called Agents of Alienation, about the danger that one day computer networks would have these automated agents, what we call bots now, which could be used to manipulate advertising and politics, and everything would become unhinged.” Trump, he suggests, is a symptom of that prophesied reality. “Going back to the 80s and 90s… there were these vivid arguments about the nature of truth made by people I like and respect, like Richard Dawkins and Daniel Dennett. Their argument was that we already have the basic outline of understanding how thought works, and so we can create algorithmic systems that capture knowledge, that exercise wisdom. I thought that was a huge trap – that it would turn into this house of mirrors that could be manipulated by whoever was the biggest asshole.”

Given that we have arrived at something like that future, what does he believe our relationship should now be with the reality-creators in Silicon Valley? “My inclination is to say that people should become acquainted enough with what the technology can do so that they are less likely to be fooled by it. If you have learned a little bit of magic, you are less likely to be tricked by a magic show, but you still might enjoy the performance a lot.”

And does he still believe that Virtual Reality worlds – he was a fan of early versions like Second Life – can take us out of ourselves, be a civilising force?

“There is a funny thing when a new medium shows up,” he says. “It takes a while to find itself. Early cinema is precious, but we couldn’t watch it now. With virtual reality, I think there are cases where it has already demonstrated its value, in treating post-traumatic stress, for example, or helping people overcome addictions. I think it will eventually become a proper medium of art and culture, but you can’t put a schedule on that.”

On the one hand, I say, his book is an involved technical insight into those futuristic possibilities, but it also reads a bit like an old-fashioned myth of seeking out new worlds and finding them at home. Towards the end of the book, settled with his wife and young daughter, he admits to something that hasn’t been so apparent in the questing that has gone before: happiness. Did he come to the conclusion that real life was more important than other worlds?

Lanier traces his impulse to seek ways of communicating back to the death of his mother in a car crash, when he was a child. Photograph: Saroyan Humphrey/The Observer

“That priority was always clear to me,” he says. “Virtual worlds can be a part of real life, but this notion that they could be on an equal footing is really abhorrent to me.”

At the very end of his memoir he writes about his father, Ellery, dying in 2014. Given that it seems Ellery set a lot of his ambition in motion, I ask him what his dad made of his achievements. He chokes up on the phone, before answering.

“It is not a conversation we had,” he says. “I think he was proud of me. And I think he was relieved, because you know, when you have a weird kid, who knows what is going to happen? But I don’t know. As a parent, though, I have found that a poisonous thing is to try to impose your own hopes on your child. I have been haunted wondering if I would have pleased my mother, but it is so hard to imagine what she would have been like had she lived…”

We’re at the end of our allotted hour on the phone and it seems a good point at which to close. Before he goes, Lanier says, pointedly, that he wants to note that we “haven’t really talked about virtual reality, which is the theme of my book…”

I’m surprised he thinks this, and I make noises about how I’m not a specialist and wouldn’t feel qualified to challenge him on the specifics of the science. But also, in my mind, I feel we have talked of little else.

• Dawn of the New Everything: A Journey Through Virtual Reality by Jaron Lanier is published by Bodley Head (£20). To order a copy for £17 go to bookshop.theguardian.com or call 0330 333 6846