The ancestors of today's gentle giants of the ocean were equipped with the razor-sharp teeth of a fearsome predator and could have hunted seals and penguins, rather than the tiny krill they eat today, scientists say.

Key points: Two groups of whales exist today — Mysticeti (modern baleen whales) and Odontoceti (toothed whales)

Two groups of whales exist today — Mysticeti (modern baleen whales) and Odontoceti (toothed whales) Research suggests ancient whales had teeth similar to those of a modern lion or dingo

Research suggests ancient whales had teeth similar to those of a modern lion or dingo Findings raise questions about how this group of whales developed baleen, the fibrous filter system they use to feed — enabling them to become the largest animals on Earth

The findings feed into the puzzle about how modern Mysticeti whales, which include blue, humpback and right whales, came to evolve bristle-like structures — baleen — that allow them to filter food from the water.

One enduring hypothesis has been that early whales, living about 25 million years ago, had intricate, rounded teeth that formed a sieve for scooping up prey, a method used by some living Antarctic seals.

But analysis of fossilised teeth of ancient whales, published today in Biology Letters, shows the animals' teeth were bladed — perfect for catching and cutting up prey, not for filter feeding.

"We're quite sure it must have been a carnivore, a raptorial feeder," Associate Professor Alistair Evans, a co-author on the study, said.

"It hopefully puts to rest one of these alternative hypotheses about how these whales evolved, and makes clearer the scenario that we think really happened."

The baleen of a humpback whale ( Flickr: NOAA Photo Library (CC-BY-2.0) )

3D analysis of old teeth

To work out whether ancient whales were predators or filter feeders, the scientists created 3D models of the teeth of the eight extinct whale species.

They also created 3D tooth models of four land-dwelling carnivores — the coyote, the lion, the dingo and the puma — and five seals (some of which filter-feed with their teeth).

"We wondered which teeth are the fossil whale teeth more like: the seals that filter feed, or the carnivores that live on land and definitely use their teeth to cut up food?" Dr Evans said.

The team analysed these tooth models for features like how sharp they were, how pointy the tips were, and whether they were notched to allow the animal to trap and cut food like a pair of secateurs.

They found that two species of seal — leopard and crabeater — were clearly distinct from the rest of the group analysed. Those species had blunt, rounded teeth that fit in with their known use of filter-feeding, Dr Evans said.

The jaw of a crabeater seal, which filters food and water into its mouth through the teeth ( Supplied: Museums Victoria )

But the whale teeth were far closer to terrestrial carnivores like the lion than they were to those seals.

That suggests early whales were "raptorial feeders" — they grabbed larger prey and chomped it up, Dr Evans said.

"You'd grab it, cut pieces off with your teeth, shake it even. Almost all carnivores you think of are raptorial."

These ancient whales may sound similar to modern toothed whales such as the killer whale, Orcinus orca.

But though the whales' diet would have been similar, Dr Evans said there are a number of important differences between the species.

"All of the modern toothed whales, including killer whales, have simple teeth — they're basically just single pegs, like a cone," he said.

In contrast, the research shows ancient whales had a number of blades and bumps on each tooth — a "mountain range" as opposed to the killer whale's "single peak."

The ancient whales would have been smaller than these killer whales, but had sharper teeth. ( Submitted: Rebecca Wellard )

The other difference is in size.

"A modern killer whale is relatively large compared to many of these ancient whales," Dr Evans said.

It was only after the whales adapted baleen, and began to bulk-feed on krill, that they ballooned in size. Modern blue whales reach up to 30 metres in length and can weigh more than 170 tonnes, the largest animal on the planet.

Where did baleen come from?

Ewan Fordyce, a professor in palaeontology at the University of Otago in New Zealand, said the research helped discard old hypotheses that ancient whales filter-fed.

"I think it's very helpful in allowing us to discard some old hypotheses, it's just great," he commented.

But while the research appears to have solved one puzzle, Professor Fordyce said it raises other questions about precisely when and how whales evolved to have baleen.

One school of thought argues that whales developed baleen that co-existed with their teeth. The fibrous filter then eventually became their predominant feeding mechanism.

But Dr Evans said it was more likely the teeth and fibrous filter existed separately.

A fossil skull of the ancient whale, Janjucetus ( Supplied: Ben Healley/Museums Victoria )

"Our fossil finds and this new research suggests first they had teeth, they became suction feeders, they lost their teeth, and then they evolved baleen to help them retain the small foods they were catching by suction," he said.

Suction feeding fish pull large volumes of water into their mouths, and food along with it, to feed. Both sperm and beaked whales are suction feeders, though they also possess teeth.

Professor Fordyce agreed with that analysis, but said more work needed to be done to find fossil specimens demonstrating that transitional period.