Ida, the fossil of an early primate, caused a sensation when she was unveiled last month. But she is just one of thousands of beautifully preserved ancient creatures that are being unearthed from an old shale quarry in Germany. Patrick Barkham pays a visit

One day, in an ancient land, a small bat began an incredible journey through time. Dusk was falling in the forest when he swooped low over a lake. Suddenly, a belch of noxious gas from a fissure caused him to fall into the water. Escaping the attentions of small crocodiles, the bat sank nearly 300m to the bottom of the lake. Gently, he settled into the sediment, as if his mother had tucked him into bed.

Over 48m years, continents shifted, the lake silted up and the little bat slept on. Much later, there was the banging and crashing of new arrivals. One of these excitable primates discovered shale oil in this small patch of German forest and dug up the sediment. The remnants of mysterious prehistoric beasts crumbled into pieces like the oil shale around them. The bat had some near-misses. When the human primates fought, pilots looked down on the giant smoking crater of the open-cast mine and assumed they had already bombed it. Later, these quarrelsome humans decided to fill its 60m crater with rubbish. Others protested and frantically dug up fossilised anteaters, tapirs and miniature horses before they could be buried again.

Finally, last week, the bat was gently prised from his stone covers by a palaeontologist, and bathed in water so he would not crumble into dust. Grime was gently brushed from his wings and tiny ears by the team of scientists led by Stephan Schaal from the Senckenberg, the largest natural history museum in Germany. The bat will be attached to a slab of yellow resin and given a name, and then he will be admired in a museum for the short time these humans walk the earth.

When a rather different fossil, Darwinius masillae, or Ida, was unveiled in New York last month, she caused a sensation. She was hailed in this paper as "a crucial 'missing link' between our own evolutionary branch of life and the rest of the animal kingdom". While the scientific significance of this celebrity fossil primate will be debated for years, her story was certainly compelling: a broken wrist, a fatal fall, millions of years in the ground and 20 in the clutches of a possessive private collector before she finally came into public view. Equally amazing, however, are the stories of thousands of other creatures who, like Ida, lived, died and have found an afterlife at Grube Messel, or the Messel Pit, one of the richest repositories of fossilised life in the world.

Unlike Ida the fossil rock star, the Messel Pit is unassuming in the extreme. Tucked behind an industrial estate 18 miles south of Frankfurt, it is badly signposted and hard to find. Local office workers shrug their shoulders; they don't know where or what it is. Down a rough lane, between a building site and lorries plundering old heaps of shale, are a couple of portable cabins for a trickle of visitors. A housekeeper doubles as a security guard. Beyond a modest fence slumps a spectacularly large crater, 60m deep and feathery around the sides with birch trees, elder, nettles and ragwort. At the bottom is a tiny pond from which rises the croak of dozens of frogs.

The frogs would have been croaking 48m years ago, when Messel was a deep volcanic lake in the subtropical rainforests of the European archipelago. While lemur-like primates such as Ida clambered through the trees, tapirs, anteaters and primitive hedgehogs snuffled in the forest. A miniature horse, 50cm high and pregnant, stood by the water. Woodpeckers pursued giant ants, crocodiles snapped at perch, turtles mated and a snake that would come to be named after the German Green politician Joschka Fischer weaved along the shore.

Turning up just 2m years ago, humans tend to look pretty insignificant in the history of the Messel Pit, but Fischer is one person who played a small part in ensuring its survival. Another is Schaal, who when not cleaning up newly discovered bats must wonder how Ida will affect the future of Messel. Each discovery triggers dozens of new questions. Why did species such as tapirs become extinct in Europe? Or how did a migratory eel get into the lake? "It's like a puzzle, every year a new piece," says Schaal. The unveiling of Ida, however, poses some less scientific riddles. Will Ida cause the pit, which is open to the public but with excavation restricted to scientists with permits, to be swamped with visitors? Will there be a return to the illicit plundering of its treasures? And is there another Ida, or something even more spectacular, still buried?

The first crocodile was pulled from the earth at Messel by miners in 1875, around the time the extraction of oil shale, used to produce paraffin and later, industrial oils, began in earnest. Oil shale is a dark, flaky sedimentary rock. Layers of shale come apart like the pages of a book, exposing rather flattened fossils, often in two halves. These fossils were amazingly well-preserved. Because the volcanic lake was so deep, no oxygen circulated at the bottom, and animal carcasses could not be eaten by scavengers. Slowly, bacteria ate the soft tissue, creating a cast of the creature through the mineral waste they left behind. As well as bones, you can see outlines of animals, the markings on its feathers, soft skin and even stomach contents.

There was one problem: these beautiful fossils were as ephemeral as a sunset. When shale is dug from the ground, it quickly dries, turning brittle and crumbly. Fossils would simply shatter. In the 1960s, however, a "transfer" technique was developed. Resin was applied to the fossil found on a slab of shale and the bones and the tracings of its soft tissue would attach themselves to the resin. "With the transfer method it was possible to get aesthetically pleasing fossils and significantly interesting fossils," says Norbert Micklich of the Hessen state museum in nearby Darmstadt (Micklich's museum and the Frankfurt-based Senckenberg are only institutions currently licensed to excavate the pit). Without this method, says the gruff Micklich, "even Ida would have looked like bullshit".

Scientists and amateur collectors quickly mastered the technique that could turn Messel's fossils into collectable treasures ideal for hanging on a living room wall. The mining may have destroyed thousands of great fossils but, as Schaal points out, "on the other hand, we wouldn't have the hole. We wouldn't be in those layers, we would be 60m above and we wouldn't have a single fossil."

The mining stopped in 1971 and Messel became a mini El Dorado. At times, 300 fossil-hunters would be digging at random. "It was like a gold rush. Many people who were not well equipped went into the pit," says Micklich. Exotic primitive relatives of tapirs and anteaters were dug up. So too were tiny horses (one with the bones of a foal still visible in its womb). In 1975, the first primate was discovered. It was a male: among the remains was the baculum - the penis bone that is found in most mammals.

Many fossil sites serve up one, characteristic kind of fossil. Messel boasts an unusual diversity: the lake was, in effect, a deadly trap for not just mammals but birds, fish, plants and even insects - brilliant beetles in iridescent orange and metallic turquoise - from 48m years ago. "It's like you open a window and look into a time you don't know," says Schaal. Scientists are building up an amazingly detailed picture of this ancient ecosystem. In Messel's fossils, they can calculate the frequency a bat echo-located on from the shape and size of its ears. Queens and kings of a species of giant ant with a 12cm wingspan have been found but, oddly, no worker ants. Sonja Wedmann, a scientist at Senckenberg, believes the queen's wings would have dragged her under when she fell on the water: humble worker ants, unencumbered by wings, could have scampered across the water to safety. Turtles the size of dinner plates have often been picked up in pairs, a large one nestled with a smaller one, back-to-back. These perished in a moment of vulnerable ecstasy: just as they mated.

In the 70s and 80s, at least some part of the febrile excitement around the pit had little to do with a spirit of scientific inquiry. Nevertheless, even the greed of private collectors had an altruistic flavour when it was the announced the pit would become a rubbish dump. Every state political party supported the landfill plan, except the Green party (and Fischer). So "emergency excavations" began. Museums were permitted to excavate fossils but illegal digging continued too: as Schaal says, people felt it was morally justifiable because it would soon be buried in garbage. The museums welcomed help from amateurs. In haste, much may have been lost. "Everyone tried to get out large mammals. Insects and smaller things were discarded," says Micklich. In the midst of this chaos, in 1983, someone found Ida. She came out in two halves. Whether they were kept together at first is a mystery but secretly, and very expertly, Ida's better half was preserved in resin.

And so, another riddle at Messel was created: how could someone whisk away the pit's greatest discovery from under the noses of everyone? Relatively easily, say the scientists. The finder would have needed people to help extract Ida but assistants could have been oblivious to what they were helping with: like most discoveries, before she was cleaned up and examined, her significance may not have been apparent. Schaal began working at Messel in 1984. He joined the campaign to save the pit. Every time he found a new fossil, he raced to the local paper and urged them to write stories about it. "The first years were really hard for me. I got my grey hairs then," he sighs. In 1987, a local court threw out the landfill plan but legal battles continued until 1991 when the state of Hessen bought the site. Four years later, Unesco crowned it a World Heritage Site.

In Germany, the owner of the land where a fossil is found owns half that fossil. Rumours of a spectacular discovery swirled among those who worked at Messel for years, but the first step towards the unveiling of Ida came with an amnesty: finds taken before it became a World Heritage Site were declared legal. "At this moment, private collectors began to show them to scientists, and sell them," says Schaal. "It was a very important step for the science."

Schaal and his colleagues at Senckenberg cultivated relationships with private collectors; gradually he built up an idea of who had what. He refuses to judge private hunters. "You can say it was not legal but it was helpful that they worked here in the pit. If they were able to prepare them, they were able to produce good fossils," he says. His contacts led Schaal to be offered Ida before she was sold. He calls her "Part A" because "Part B", the other, less perfect half of Ida, was identified several years ago and purchased by Burkhard Pohl, a German vet turned collector who placed it in his private museum in Wyoming. (Part A was eventually bought by the palaeontologist Jørn Hurum, the man who paraded her before the press last month.) "Senckenberg knew about Part A two years ago and they asked if we wanted to buy it and it was too expensive," says Schaal. "Of course, museums like to have such a piece but we decided not to buy it, so we don't regret it now. This find is important because it's complete and because the quality is good. It is good that it is public but I cannot say more about the real significance about whether it's the missing link or not. The main point is that science has now got the work. Everybody can look at it and study it. This is what is important."

The unspoken implication is that Ida is not as significant as billed, but Schaal won't be drawn. It is not sour grapes: two of what he calls, without irony, the "dream team" of scientists who wrote the academic paper on Ida are from Senckenburg. Micklich seems rather more unimpressed by the hullabaloo around Ida. He also looked at Ida around the time she was sold. "We have seen this specimen," he says of Ida. "It is a counterpart of a described specimen [Part B, the less good half of Ida, in the Wyoming museum] and we wondered how could a known species now be an unknown species? But we could not imagine this hype." His colleague, Gabriele Gruber, tactfully adds: "It's a very fine fossil. It is very well preserved."

Will Ida help Messel? An exhibition of remarkable Messel finds from Micklich's museum is currently touring Germany after museums in London and America turned it down. They probably would not do so again. Micklich and Gruber agree that Messel's raised profile may help its scientists find more financial support. Then again, as scientists have a habit of saying, it may not. Overhyped palaeontology that panders to our fascination with mammal fossils creates a problem for the likes of Micklich and Gruber, who are studying fish and invertebrates. "It is dangerous because our work is evaluated by these striking records," says Micklich. "If you can't hold a striking press conference every year then people will ask, 'Why are you costing us this money?'"

Are there more spectacular finds buried in Grube Messel? During the April-September excavation season, every day, on average, Schaal's team of a dozen researchers will find 10 fish, 10 plants and 10 insects. Every week they will find a bird or a bat. Schaal is confident there will be more Idas. "Every year we find more big mammals. It is possible we will find the next primate tomorrow, or in 20 years, but as we are digging every summer the chances are quite good. It's only a question of time before we find the next primate".

• For more on the significance of the Ida discovery, including a video with Sir David Attenborough and an interactive guide to the fossil, go to theguardian.com/fossil-ida