Two years ago he worked on Jay-Z and Kanye West's Watch the Throne tour, overseeing a web operation called Voyeur that sold fans behind-the-scenes videos from the previous night's show. It was a gruelling, non-stop production and editing job. So when he arrived in Tulum, on the Mayan Riviera, after the tour ended, he slept around the clock, missing an entire day. One morning, he awoke with vivid memories of a dream in which he climbed the spiral staircase at the Reichstag in Berlin, in the company of Michael Jordan. He realised that it had been a long time since he was last able to recall a dream and resolved to both sleep better and pay more attention. Soik had not stopped dreaming – the discovery of rapid eye movement (REM) sleep in 1953 established that almost everyone experiences multiple dreams each night – they just failed to imprint on his consciousness, like a roll of negatives exposed to the light. Shadow aims to commit dreams to memory, something that we're not very good at doing by ourselves. Using a motion sensor, the app monitors the user's sleep cycle, to wake her at the optimum moment for dream recall, shortly after a period of REM sleep. An alarm gradually increases in volume, a recorded voice asks "What were you dreaming about?" and a recorder is activated automatically, so that the dream can be dictated. Voice recognition software converts to text, and an algorithm pulls out keywords, to be logged.

The report can be stored locally, shared with friends or uploaded to the cloud anonymously, where other members of the Shadow community can comment. Kelly Bulkeley, a dream researcher and consultant on the project (Soik refers to him as "one of our six PhDs") compares it to a dream discussion group. "Any system that says 'we're gonna tell you what your dream means', run in the opposite direction," Bulkeley says. "But hearing other perspectives triggers new awareness of your own experience. That's the hope, the promise and the potential of Shadow." The idea that our dreams are telling us something is as old as humanity. The chamber at Chauvet cave in southern France, with its prehistoric paintings of horses, panthers and bears, may have been a site where hunters sought guidance from dreams. In Homer's Odyssey, Penelope suggests that there are "two gates" through which dreams proceed. "Those that come through the gate of ivory are fatuous, but those from the gate of horn mean something to those that see them." Aristotle wrote about prophetic dreams at length, in scepticism, but not without sympathy. In the Bible, for every divine instruction revealed to Jacob, Joseph or Paul as they sleep, there is a warning that dreams are "dark speech" and cannot be trusted. Shadow is presented as a means of unlocking these mysteries, both at a personal level and by assembling an unprecedented database for researchers. "There's huge debate on the function of dreaming. And nobody can give you a clear answer," says Soik. "We're explorers. We're going to the moon and we're putting a webcam on the spaceship." The Interpretation of Dreams, by Sigmund Freud, remains the most influential text in psychoanalysis, although many of its assumptions have been disproved in sleep laboratories. Freud viewed dreams as "the royal road to a knowledge of the unconscious activities of the mind".

His estranged disciple Carl Jung placed less emphasis on repressed aggression and sexual desire, but agreed that in the course of professional interpretation "every dream will reveal itself as a psychological structure, full of significance." In the last half-century, neurophysiologists have challenged Freud and Jung. Harvard professor Allan Hobson asserts that dreams are essentially meaningless: the forebrain attempting to make sense of signals fired by the brain stem as best it can. Finnish researcher Antti Revonsuo's "threat-simulation theory" posits that the purpose of dreams is to play out dangerous scenarios in a safe environment, while British molecular biologist Francis Crick's "reverse learning" theory suggests that they flush the brain of unnecessary information. None of these theories has anything like the popular appeal of Jung's description of dreams as a "theatre in which the dreamer is himself the scene, the player, the prompter, the producer, the author, the public and the critic". And indeed, recent research at the University of Rome, showing that the brain retrieves and encodes memories in much the same way asleep as it does awake, suggests that psychological explanations may be closer to the mark. At present, dreambank.net is the world's largest academic repository of dreams, with just short of 30,000 on file. Shadow aims to recruit 2 million users in the first year. "There are a lot of studies that benefit just by having very large numbers," says Deirdre Leigh Barrett, a professor of dream research at Harvard University who has consulted on the project. "Having access to that many dreams is exciting." As Soik pitches it, Shadow will open up new lines of inquiry and settle old arguments: How do dreams differ from country to country? Are certain nightmares universal? How much do current events influence dreams? How does the content of our dreams change as we age? We already know that women are much more likely to record and discuss their dreams than men. Will Shadow change this?

One problem faced by dream researchers is what's known as the "lab effect": sleeping in a laboratory, wired up to machines, has a profound influence on dreams. Study participants experience far fewer nightmares, for instance. Shadow skirts this problem, but runs into another: knowing that their dreams will be discussed, users have an incentive to exaggerate them or make them up entirely. "Many people keep dream journals for their own purpose, and that's what the app is designed for, but even then, there are people who might choose not to write down a particular dream because it's embarrassing and shameful," Barrett says. "And when you're recording it for other people, how much it's socially accepted has got to influence content." Bulkeley concedes that "there are limitations to this kind of data, but nevertheless it's an exciting prospect. Suddenly we've got vastly more dreams at our disposal." In the light of recent disclosures about the US security sector's voracious appetite for personal information, I ask Soik, half in jest, how long it will be before the National Security Agency is tapping into our dreams. "We take user privacy very seriously. Everything is encrypted on the transmission from the phone to the server," he replies, in earnest. Shadow's servers will be in Iceland, Germany or Switzerland, or other countries with strong data protection policies. The dream data would also be extremely valuable to advertisers, as a collection of unconscious desires to be pandered to and exploited. "It's pre-cognitive," Soik points out. "When you think about what we're collecting it's before Google, it's before Facebook, it's the subconscious that creates." But he insists that the dreams archived by Shadow will never be sold. "We're not about ad-based revenue. We're trying to create something that is an educational platform for self."

I wonder how it will make money, but as always, Soik is ready with an answer. "Sleep is a $US32.4 billion [$35.4 billion] industry," he says. "It grows at 8.2 per cent per year: $US11 billion is medical, $US11 billion is mattresses and $US11 billion is soft goods. I think we can tap all three markets." If the idea of Shadow-endorsed sleeping pills and branded sheet sets isn't entirely convincing, perhaps it doesn't matter, in a market in which Twitter is valued at $25 billion on the first day of trading, despite revenue of just $535 million. To make it work, Shadow must convince millions of people that it is worth systematically recording their dreams: that there is insight and inspiration to be found without the help of a professional analyst. "There's all sorts of things that one can learn by analysing the symbols in dreams, but I've often tried to persuade people that, apart from that, there's also value in bringing dreams and their images, feelings and experiences into waking awareness," says Bulkeley. Einstein once said his theory of relativity was inspired by a dream in which he was sledding down a steep slope, observing how the appearance of the stars changed as he approached the speed of light. Frankenstein came to Mary Shelley as she slept. Robert Louis Stevenson attributed his best work to the "little people" that perform in "that small theatre of the brain that we keep lighted all night long". All he had to do, he said, was write down what they presented to him. These days, there's an app for that.

Age readers can get the Shadow app before its public launch by using the password 'shadow' at discovershadow.com/ambassador/.