Mammals experienced a boom in diversity long before the dinosaurs became extinct 66 million years ago, overturning the notion that their evolution was curbed by the success of the land-based reptiles, scientists have said.

It has long been thought that dinosaurs kept diversity among mammals in check by dominating food and resources, with early mammals thought to be limited to small, insect-eating creatures. But recent fossil finds have called the idea into question, suggesting that they had a wider range of shapes and sizes than previously thought.

Now scientists have shed further light on the matter, suggesting that early therian mammals – the predecessors of today’s placental and marsupial mammals – had developed a wide range of diets and ecological roles about 10 to 20 million years before the dinosaurs were wiped out.

“It suggests that the mammal story might not be as dependent on the dinosaurs as was thought,” said David Grossnickle, a PhD student at the University of Chicago, a co-author of the study. “Even if the mass extinction hadn’t occurred, maybe mammals would have continued to diversify into some of the forms we have today.”

Writing in Proceedings of the Royal Society B, researchers at the University of Chicago and the University of Southampton reveal how they turned to museum collections to scrutinise the molars of therian mammals.

The scientists found that the teeth showed a variety of shapes, suggesting that the mammals had a range of diets and an assortment of ecological roles. “The great thing about molars is they are reflective of diet. Sharp, blade-like molars are going to indicate carnivory – meat-eating diets – whereas broad, flat molars are more indicative of plant-eating diets,” said Grossnickle. “The early therians were probably insect-eating small creatures like small rodents or shrews, but right before the [mass extinction] there is definitely evidence that there are some plant-eating therian mammals and some more carnivorous therian mammals.”

The authors believe this increase in diversity could be linked to large evolutionary changes among flowering plants that took place during a similar period – a suggestion that has previously been proposed to explain the increase in diversity among another group of rodent-like mammals known as multituberculates that also occurred during this period.

“The increase in flowering plants likely increases the number of available seeds and fruits – also insects might be co-evolving with the plants,” said Grossnickle. “So you have more seeds, fruits and insects that the animals could have eaten.” But, he adds, the idea is only a suggestion at present. “It is very speculative,” he says.

While therian mammals survived the mass extinction 66 million years ago that wiped out dinosaurs, with the exception of birds, the findings reveal that their diversity was affected. That, the authors say, appears to be down to the extinction of mammals that had highly specialised diets. “Things that seem happy eating more or less anything – we see more of those after the [mass extinction] than before,” said Grossnickle. “That tells us that maybe it was better to be generalised in terms of surviving the mass extinction event.”

Grossnickle believes the new research could also tie in with the much-debated suggestion that dinosaurs were in decline long before the asteroid struck, triggering the mass extinction. “If you believe that dinosaurs were on the decline, maybe in part because of environmental or climate changes, then you might also predict that mammals should have been on the decline,” he said.

“On the other hand, you could also predict that mammals would have taken advantage and started to diversity as dinosaurs vacated different environments or niches. This would be in line with our results.”

Others are more sceptical. “Dinosaurs by and large are pretty massive. Mammals in the interval preceding the extinction are much smaller in comparison, and it is hard to imagine that they had ecologies broadly equivalent to the dinosaurs they lived alongside,” said Matt Friedman, a palaeobiologist from the University of Oxford.

But Friedman agrees that the new research contributes to a changing view of early mammals. “We do have this stereotypical view of mammals in the Mesozoic under the foot of dinosaurs, which from the perspective of size they were,” he said. “But those small mammals actually occupied a surprising number of ecological roles. I think this adds to that growing picture.”