The sperm whale – Physeter macrocephalus – is an oddball among living cetaceans. As big as many of its baleen-bearing cousins, yet armed with a lower jaw full of teeth slung below its blunt head, the sperm whale is the largest predaceous vertebrate in the sea.

It might have been cast as a gentle giant by various save-the-whales organisations but, while it is certainly worthy of protection, it was whalers of centuries past who knew full well the power of this marine mammal. Their impressions of the cachalot were immortalised in literature by Herman Melville's Moby Dick. Less an animal than a force of nature, Melville's antagonist was the "monomaniac incarnation of all those malicious agencies which some deep men feel eating in them". But even an enraged bull sperm whale would have paled in comparison to some of its fossil relatives.

During the middle of the Miocene epoch, between about 13m and 12m years ago, the sea which once blanketed present-day Peru was home to an array of marine predators of gargantuan proportions. The famous "megashark" Carcharocles megalodon swam these waters and so did the prehistoric cousins of the modern sperm whale, called physeteroids. This group of ancient sperm whales is only represented by three species today (the sperm whale, the dwarf sperm whale, and the pygmy sperm whale), but the sperm whales of the Miocene were quite distinct from their extant cousins.

In July, a team of paleontologists led by Institut Royal des Sciences Naturelles de Belgique scientist Olivier Lambert described one of the ancient beasts in Nature.

Livyatan mellvillei certainly would have lived up to its name*. With a head almost three metres long, this 12m-year-old-physeteroid was about the size of a bull sperm whale and had wide, v-shaped jaws studded with conical teeth in both its upper and lower jaws. This whale had one of the largest bites in the history of life on earth, and even the paper's scientific description of its probable feeding habits verge on the sensational: "[Livyatan had] a short and wide rostrum allowing a more powerful bite by the anterior teeth and better resistance to lateral movements of the struggling prey, [and] procumbent anterior teeth for grasping voluminous prey with moderately convex body surfaces … This sperm whale could firmly hold large prey with its interlocking teeth, inflict deep wounds and tear large pieces from the body of the victim."

In other words, the robust skull of Livyatan was well-suited to dealing with the physical stresses involved in catching, killing, and consuming the baleen whales which also inhabited the waters of prehistoric Peru. More than that, as a marine mammal Livyatan would have maintained a high, constant body temperature, and therefore would have required an enormous amount of energy to fuel its body. Pound-for-pound, other whales were richer in fat and other high-energy resources than smaller prey in the area; for a large predatory whale, there was no better meal than another whale.

The feeding choices of Livyatan – as well as the giant shark Carcharocles megalodon – may have even influenced the pattern of whale evolution. That both large predators were present in the same place at the same time is a testament to the availability of prey, and the middle of the Miocene was a time when whales were both diverse and prolific. Lambert and colleagues think that large predators may have partially driven these trends, particularly increasing body size in prey animals (the bigger you are, the less likely you are to fall victim to predators).

This correlation requires more evidence to understand how predation may have affected cetacean evolution, but the occurrence of many large teeth from physeteroid whales in deposits of similar age around the world suggests that, during the Miocene, both "killer sperm whales" and large sharks were important components of marine ecosystems, just as orcas and great white sharks are today.

That whales like Livyatan even existed is still a relatively recent discovery. For decades paleontologists had found the fossil teeth of physeteroid whales, but it has only been within the past few years that scientists have been able to recover more diagnostic remains from these predatory whales. In 2008 Lambert and a different group of scientists described another toothy physteroid whale from Peru they named Acrophyseter deinodon. At 6m years old and approximately 5 metres long, it was a smaller ecological successor to Livyatan. Still, it was probably the apex marine predator of its time. Like its elder cousin, Acrophyseter had a wicked smile full of teeth like recurved railroad spikes, and it probably fed upon the small baleen whales, seals and penguins which shared its habitat.

Killer sperm whales were not just restricted to the waters of prehistoric South America. In 2006 researchers Toshiyuki Kimura, Yoshikazu Hasegawa, and Lawrence Barnes reassessed 15-14m-year-old fossil remains from Japan which had been attributed to a little-known physeteroid called Scaldicetus. This genus had been established on the basis of a few isolated teeth from Belgium, but when further fossil remains of the whale from Japan were found they were distinctive enough to establish the new genus of killer sperm whale, Brygmophyseter.

The ocean of ancient Italy had its own killer sperm whale, too. Described by Giovanni Bianucci and Walter Landini the same year as Brygmophyseter, the eight-metre-long Zygophyseter varolai was one of the strangest fossil whales yet discovered. The head of Zygophyseter was not as orca-like as those of Acrophyseter and Livyatan, but instead was elongated with a very deep cranium. In profile, it would have looked like a dolphin-sperm whale hybrid, with a thin snout in front of a massive, blunted mass which probably housed a spermaceti organ – a fatty collection of tissues which are of uncertain function in living sperm whales.

Through their discovery and description, Brygomophyseter, Zygophyseter, Acrophyseter, and Livyatan have fleshed out the wider evolutionary context for the extant sperm whale. The cachalot is the suction-feeding remnant of a lineage of rapacious, predatory whales which were the equivalent of orcas in their ecosystems, and we now know that some of the traits previously thought to be unique adaptations of the living sperm whale are actually many millions of years old.

One of the most peculiar aspects of the living sperm whale is a mass of tissue cradled on its skull called the spermaceti organ. Sitting on top of a lump of tissue called the "junk", this oil-filled sac was prized by whalers, but its actual function in sperm whales remains unknown. Perhaps it helps the whales regulate buoyancy during deep dives, or maybe it aids in echolocation. Then again, Melville may have been right that the organ evolved as a battering ram for male-male competition – which also proved useful in sinking whaling ships – but whatever its function the spermaceti organ has typically been believed to be an adaptation to some peculiarity of the sperm whale's natural history.

When Physeter macrocephalus is viewed in the context of its prehistoric cousins, however, the anatomy of the fossil physeteroids shows that the spermaceti organ was present in the predatory sperm whales first. These whales were not deep-diving squid suckers. Instead, the killer sperm whales were fast predators which took devastatingly large bites out of their prey, and therefore it is possible that the spermaceti organ originally evolved for reasons unrelated to its present use. If this is correct, then the spermaceti organ became co-opted to perform new functions over time; even if we eventually understand what living sperm whales are using the organ for, its current function may not tell us why it evolved in the first place. That is part of the beauty of evolution. Gradually, old traits are repurposed as populations become modified over generations, creating the "endless forms most beautiful and most wonderful" which left Charles Darwin in awe of nature.

*(This whale was named Leviathan, but, unfortunately, that name had already been applied to another fossil mammal and so could not be used again. In the mid-19th century the German paleontologist and showman Albert Koch applied the name Leviathan to an American mastodon [Mammut americanum] skeleton he had artificially enlarged by adding a few extra bones. It didn't matter that Koch was trying to rename an animal which had already been given a scientific name; Leviathan was relegated to the dustbin of taxonomy and Lambert et al chose to rename their whale Livyatan.)

Brian Switek blogs at Laelaps