The famous Kalahari Bushmen of southern Africa have long been in decline. For more than a century, the people, who speak Khoisan languages, have been pushed off their land by farmers and brutalized by colonialists.

Yet for tens of thousands of years, the Khoisan’s ancestors were members of “the largest population” on the planet, according to a new study.

The Khoisan have long stood apart from other groups within Africa. They look distinct, speak in “click” languages, and have also maintained the greatest genetic diversity known among human populations. Usually, big populations harbor the most diversity. But census counts show that the 100,000 Khoisan speakers in Africa today are far outnumbered by other groups, such as the 45 million Bantu speakers and their 180 million descendants who now speak Swahili and other languages. Researchers have thought that the Khoisan inherited their genetic diversity from a large ancestral population, an idea supported by a single Khoisan genome published in 2012. But scientists couldn’t rule out that the variation in Khoisan DNA arose from more recent interbreeding with other diverse Africans.

In the new study, published online today in Nature Communications, biochemist Stephan Schuster of Nanyang Technological University in Singapore and colleagues sequenced the complete genomes of five Khoisan hunter-gatherers from Namibia and compared them with the DNA from 1462 genomes of people from around the world. Schuster’s team found that two of the Khoisan, members of the Ju/’hoansi population in Namibia, inherited their DNA only from Khoisan ancestors in the northern Kalahari region and showed no sign of interbreeding with non-Khoisan speakers. These two Ju/’hoansi genomes preserve ancient diversity inherited entirely from their direct ancestors, the authors say.

Using several different methods of analysis, the team reconstructed population sizes for the ancestors of the Khoisan, as well as for Europeans, Asians, and another African group, the Yoruba. They found that all four groups declined in effective population size (the number of breeding adults) between 120,000 and 30,000 years ago. The non-Khoisan groups’ numbers plunged precipitously—by 30,000 years ago, European and Asian populations had plummeted by 90% from their peak, thanks to population bottlenecks caused by the migration of small groups out of Africa. But the Khoisan population declined by only 26%. (Yoruba populations dropped by 69%).

The researchers uncovered declines in population in all four groups, likely tied into periods of dry climate in Africa. But the Khoisan suffered the least, perhaps because their huge ancestral population was buffered from droughts that winnowed other groups, such as the Yoruba and the African ancestors of Europeans and Asians, because the Khoisan lived farther south in Africa where rainfall actually increased. The Khoisan then began a more drastic decline in the past 20,000 years or so, with a major blow when the Bantu farmers spread through Africa 4000 years ago. “This shows us how much climate can influence populations,” says genomicist Webb Miller of Pennsylvania State University, University Park, a co-author.

Other researchers agree that it’s likely that the Khoisan descend from a large population. But because sampling of African genomes is still so spotty, not everyone is yet convinced that the Khoisan “was the largest population on Earth at some point,” says evolutionary geneticist Pontus Skoglund of Harvard University. “Many African populations are not included for comparison,” he says, so it is possible that some of the diversity seen in the Khoisan was inherited from recent interbreeding that cannot yet be detected.

Either way, the study makes it clear that even though the Khoisan are genetically diverse by today’s standards, even they carry just a fraction of our ancestors’ genetic legacy over the past 120,000 years. “It is quite staggering how much extraordinary genetic variation and ethnic diversity was present but is now lost,” Skoglund says. The Khoisan, retaining more than the rest of us, offer a rare window to look back in time at some of that diversity.