Human evolution must be the greatest story never told. It begins in an unknowable past and continues mysteriously for the next five or six million years. Is it a thriller, an epic or a comedy of errors? There is no dust jacket, no title page, no dedication, no acknowledgements. Almost all the text is missing, apart from the occasional phrase, sentence or paragraph, seemingly torn at random from the great six-million-year narrative. If the story of humanity is a single volume, then only the last page survives.

Every so often, scholars find yet another fossilised scrap of the missing narrative, a new character enters, and the plot takes a new twist. Some things are clear: the story began in Africa, between 5m and 7m years ago, with the last common ancestor of two kinds of chimpanzee and of Homo sapiens sapiens. Charles Darwin calculated as much when he began telling the story in The Descent of Man (1871). "We thus learn that man is descended from a hairy, tailed quadruped, probably arboreal in its habits, and an inhabitant of the Old World," he wrote.

Anthropologists agree on the human-ape connection. The consent is there in the titles of books published in the past 40 years: The Aquatic Ape, The Naked Ape, The Third Chimpanzee, The Talking Ape, Our Inner Ape, The Thinking Ape, The Monkey in the Mirror, The Hunting Apes, The Ape that Spoke and The Artificial Ape.

These books are all attempts to work backwards, from what we are now to what we might have been. The fact that zoologists, anthropologists and palæontologists can write so many books with the word "ape" in the title tells us two things. One is that the evidence is so sparse that people are free to frame a favourite hypothesis about what it was that made humans different. The other is that the human-chimpanzee connection is so clear that there is nowhere else to begin.

First, the family likeness: chimpanzees struggle for status, vocalise, communicate, play politics, use subterfuge, show aggression, reject outsiders, groom and support each other, betray each other and resort to violence or sexual bribery to get their way. Chimpanzees display awareness of self, ability to reason, and a grasp of numbers. Chimpanzees are opportunistic omnivores that also make and use tools for gain, and groups of chimpanzees in the wild have separate traditions, practices and ways of doing things that they pass down the generations. That is, chimpanzees have culture. Chimpanzees and humans have a genetic kinship so close that they share almost 99% of their DNA.

The Victorians called them "man-like apes". Twentieth-century scientists and observers started referring to humans as naked apes. Early in the 21st century, some taxonomists and conservationists began a campaign to change the chimpanzee genus from Pan to Homo, so close are the parallels between the species.

But the African chimpanzee is an endangered species, down to perhaps 150,000, while the human population is about to tip seven billion. The implication is that, long ago, the earliest human ancestors also lived in small social groups, and co-operated and competed for the resources of the woodland and the savannah. Why did humans become so different: bipedal, upright, hairless, with limited strength, feeble jaws, bad backs, embarrassingly large heads and brains with a cerebral cortex four times the size of a chimp's?

For decades, the conventional evolutionary lineage was a simple one: shambling simian stands upright, evolves into bipedal hairy brute, then slouching hairy brute with hand axe and finally into hairless human with BlackBerry. This is the ladder theory of human evolution. It was kicked away long ago. Discoveries in Africa – a femur here, a fragment of skull there, a pelvis, now and again a partial skeleton, a set of footprints fossilised in ancient volcanic mud – reveal a picture more of confusion than direction: a flowering of creatures more or less apelike or manlike, some of them possibly direct ancestors, some of them probably cousins along a parallel lineage, all of them trying to make a subsistence living in a very different Africa, millions of years ago. The fossils turn up in South Africa, East Africa, Ethiopia and even the Sahel. They have generic names such as Sahelanthropus, Ardipithecus, Orrorin, Australopithecus, Paranthropus, and Kenyanthropus, and their remains were unearthed from the dust, stone and mud sediments laid down 3m, 4m and 5m years ago.

Two million years ago, creatures that bear the generic name Homo begin to appear in the fossil record: Homo habilis, Homo ergaster, Homo erectus, and with them appear worked stone tools, hand axes, things for chipping and cutting. Hardly any of these early human relics is complete. Palaeoanthropologists were once fond of saying that the entire human fossil record could be laid out on one table, or packed in a matching set of Gucci luggage, but this is no longer true. What is true is that even 2m years ago, the human lineage begins to look like a bush, with species sprouting in all directions.

And then the story starts to get really complicated. At some point, early humans get up and start moving. They spread. They pack their hand axes, leave Africa and start to colonise the Middle East, Europe, and South Asia. And there is more than one migration out of Africa: first Homo erectus or something even more primitive, and then, much later, Homo sapiens. And they continue to differentiate into new species. At one point in human history, around 40,000 years ago, modern humans must have shared the planet with at least four other human cousins: Homo erectus, the Neanderthals, a strange, small-brained human found only on the island of Flores in Indonesia, affectionately known as the Hobbit; and most recent of all, species X: a separate human genetic lineage identified in 2010 only by DNA extracted from a finger bone found in a Siberian cave.

What gave early humans their get-up-and-go? Why did humans develop large brains and long legs? Should the first mobile humans be classed as asylum seekers, driven from their native land by climate change? Or were they economic migrants, on the lookout for better opportunities in wide-open Europe and Asia?

Brains are what biologists call expensive items: the human brain at rest consumes 20% of the daily calorific intake. In other words, brains have to be fed. So a large, greedy brain becomes valuable only if it helps to deliver even more food and greater security. So was the larger brain a genetic mutation that increasingly delivered a selective advantage in the struggle for survival? And how did humans get from thinking about food-gathering strategies to thinking about taxonomy, tax-avoidance and Twitter?

The big brain story may have begun in the trees. Arboreal primates that search over wide areas for food in the canopy seem to know what is good for them: they often ignore easy supplies and go looking for special foods. They seem to have a notion of a balanced diet – protein-rich leaves and high-calorie fruits and not too much fibre – and they have been watched deliberately selecting plants with medicinal properties. All this requires a working memory, a mental map of where to go and what to look for. According to at least one study, the primates that hunt high and low for the quality fare tend to have larger brains than those that do not.

Then the human story begins at some point with climate change: in a cooler and more arid continent, once-arboreal creatures had to start exploiting the woodland and savannah. It would clearly be an advantage to stand up and walk on two feet, to see further, to have a hand free to carry an infant. Pair bonding – love and marriage to non-biologists – is already an evolutionary feature, and a bipedal male could go further to find food for his family, and carry it back.

"Darwin argued that bipedalism freed the hands," says Chris Stringer, head of human origins at the Natural History Museum. "He was arguing that 150 years ago and it is still there. But there is another view worth considering: it could have begun in the trees. Orangutans, for instance, walk bipedally." To get to the tastiest forage, orangutans walk along branches, holding on to yet higher branches. So there could have been a long period when early members of the not-yet-human family walked on the ground, and lived in trees.

And by this time, brain size had begun to increase. There are new challenges, new opportunities, new foods to try and new difficulties to overcome. In the past three decades, researchers have floated a number of ideas about how the human story might have developed. Did hominids start to develop bigger brains because they lost most of their body hair? A hairless human with a talent for exuding sweat would be at less risk of overheating; longer legs would enhance the surface-to-volume ratio and keep the brain cool; and as a bonus, ticks, lice and other parasites would have nowhere to hide.

Or did hominids become free to develop bigger brains because their jaw muscles began to shrink, allowing the cranium to expand? Did early humans start to develop even bigger brains because they became increasingly efficient endurance runners that could get to a carcass before the hyenas and vultures, and strip away a nourishing meal of meat, fat and marrow? Did humans begin to stand upright by taking to the water – and to nourish bigger brains with high-protein deliveries of fish and shellfish?

Did humans discover the use of fire millions of years ago, long before the colonisation of Europe? Cooking would make plants both more nourishing and easier to digest; it would dispose of infections and pathogens in meat, and it would deliver greater supplies of energy per mouthful. Teeth, jaws and digestive tracts could shrink, and so brains could get bigger. Did humans grow bigger brains because the extra neural circuitry was needed to make sense of the demands of social and co-operative life?

"I think a lot of our brain is actually mapping relationships, and mind-reading our friends and enemies: what are they doing? You need a lot of processing power to do that well," says Stringer. "If you are starting to hunt animals, you have to out-think them, and that is driving the growth of more processing power and bigger memory. So I think the social brain and meat-eating was the key to that."

Somehow, out of this million-year-mix of food, fear and hunter-gatherer companionship in Africa, complex language emerged. The human who could frame the sentence "You wait behind that rock at the end of the ravine and I'll drive the deer towards you" has demonstrated awareness of cause and effect, of geography, of zoology, of strategy, of co-operation for future mutual advantage. Somewhere in such a sentence there is also the germ of the first play for two actors, the first computer game and the first adventure story.

But there are no neat stories to be told of the first departure from the African homeland. Once again, the evidence is fragmentary, sometimes teasingly ambiguous, and capriciously rare. But there is enough to confirm the presence of early human species in Georgia, in Spain, Portugal, Germany and Britain as early as 800,000 years ago, and also in the Middle East and South Asia. The first migrants could have been pushed out of the country by climate change, or competition for resources, or the desire for somewhere new. They could possibly have made a direct crossing by water from the Horn of Africa to what is now Yemen, or they could have travelled up the Nile Valley and across what is now Gaza into Europe and the Middle East. This fabulous odyssey may not have been intended, it may have just happened. Hunter-gatherers follow game, and when the game disappears, they move on. All these first migrants needed to do was to hug the coast: first up the western shore of the Red Sea, and then down the coast of Arabia.

"They just extended in that ribbon of the coast, out of Africa, around Arabia, around the southern Asian coast: at low sea level, they could have got all the way to Java just on the coast. Then they just need to invent boats along the way and they can get to Australia," says Stringer. "One mile a year and you have gone all the way to Java in 10,000 years."

And in the course of this great adventure, the migrants change. New species appear, and with them, new behaviour. The Neanderthals become the first to formally bury their dead.

And long afterwards, modern humans turn up. Once again, the story begins somewhere in Africa, nobody knows for sure where, and once again, at least 60,000 years ago – and maybe, on recent enigmatic evidence of stone tools in Arabia, as long as 125,000 years ago – a new human species begins to leave Africa and spread around the planet, across all of Europe and Asia, and then finally across the arid freezing plains that will in time become the Bering Straits, to Alaska and then the whole of the Americas. Modern humans are still hunter-gatherers, but around 30,000 years ago there is evidence of sophisticated technologies based on stone and bone and shell. They use needles, decorate with ochre, create works of astonishing art, put on ornaments, and exhibit a sense of religion – the evidence for all these things lies alongside the human fossils. In Europe, these newcomers live alongside the Neanderthals, hunt the same animals, gather the same seeds and fruits. There is recent evidence that – somewhere in the European chapter of this story – modern humans and Neanderthals must have interbred, but in all other respects, the Neanderthals seem to be a different species.

Long before the end of the last ice age, the Neanderthals and all the other human species that have travelled the same road vanish altogether, leaving the newcomers alone of their kind, and in undisputed possession of the planet.

Tim Radford's geographical memoir The Address Book: Our Place in the Scheme of Things, is published by Fourth Estate on 28 April at £16.99. To order a copy for £13.59 with free p&p, call 0330 333 6846 begin_of_the_skype_highlighting 0330 333 6846 end_of_the_skype_highlighting or visit theguardian.com/bookshop