Perhaps, when we’re very young, the hippocampus simply isn’t developed enough to build a rich memory of an event. Baby rats, monkeys and humans all continue to add new neurons to the hippocampus for the first few years of life and we all are all unable to form lasting memories as infants – and it seems that the moment we stop creating new neurons, we‘re suddenly able to form long-term memories. “For young babies and infants the hippocampus is very undeveloped,” says Fagen.

But is the under-formed hippocampus losing our long-term memories, or are they never formed in the first place? Since childhood events can continue to affect our behaviour long after we’ve forgotten them, some psychologists think they must be lingering somewhere. “The memories are probably stored someplace that’s inaccessible now, but it’s very difficult to demonstrate that empirically,” says Fagen.

We should be very wary about what we do recall from that time, though – our childhood is probably full of false memories for events that never occurred.

Elizabeth Loftus, a psychologist at the University of California, Irvine, has devoted her career to the phenomenon. “People can pick up suggestions and begin to visualise them – they become like memories,” she says.

Imaginary events

Loftus knows first-hand how easily this happens. Her mother drowned in a swimming pool when she was just 16. Years later, a relative convinced her that she had discovered her floating body. It all came flooding back, until a week later the same relative called and explained she’d got it wrong – it was someone else.

Of course, no one likes to be told their memories aren’t real. To convince the sceptics, Loftus knew she’d need unequivocal proof. Back in the 1980s, she recruited volunteers for a study and planted the memories herself.

Loftus spun an elaborate lie about a traumatic trip to a shopping mall when they got lost, before being rescued by a kindly elderly woman and reunited. To make the event more plausible, she even roped in their families. “We basically said to our research participants ‘we’ve talked to your mother, your mother has told us some things that happened to you.’” Nearly a third of her victims fell for it, with some apparently recalling the event in vivid detail. In fact, we’re often more confident in our imaginary memories than we are in those which actually happened.

Even if your memories are based on real events, they have probably been moulded and refashioned in hindsight – memories planted by conversations rather than first-person memories of the actual events. That time you thought it would be funny to turn your sister into a zebra with permanent marker? You saw it in a family video. The incredible third birthday cake your mother made you? Your older brother told you about it.

Perhaps the biggest mystery is not why we can’t remember our childhood – but whether we can believe any of our memories at all.

Join 600,000+ Future fans by liking us on Facebook, or follow us on Twitter, Google+, LinkedIn and Instagram

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.