Twenty years ago, the writer and environmentalist Heathcote Williams launched an epic plea for the future of the whale. His televised poem Whale Nation – with its closing lines, "From space, the planet is blue / From space, the planet is the territory / Not of humans, but of the whale" – was a hymn to the beauty, majesty and intelligence of the largest mammals on earth, as well as a prayer for their protection. And the film was a stunning success around the world, attracting some of the largest audiences ever seen for a "nature" programme; the published version, described as a modern version of T S Eliot's The Wasteland, was a major bestseller too, with the American rights alone selling for $100,000 – a figure that prompted headlines at the time. Whale Nation became the most powerful argument for the newly instigated worldwide ban on whaling – and for a moment, back in 1988, it seemed as if a shameful chapter in human history might finally be drawing to a close.

Yet two decades on, the issues that Williams so passionately exposed are still with us. The Japanese are still hunting in the Southern Ocean – supposedly a whale sanctuary. Since 1987, when the international moratorium took effect, an astonishing 25,000 great whales have been killed. Under its Antarctic Research Programme, known as JARPA, and its North Pacific equivalent, known as JARPN, Japan has killed 7,900 minke whales, 243 Bryde's whales and 140 sei whales, as well as 38 sperm whales, which it resumed hunting in 2000. In 2006, JARPA II took 1,073 minke whales – known to their hunters as "cockroaches of the sea" – and added 50 fin whales to the tally. These cetaceans (from the Greek for "sea monster") died under the aegis of "scientific research". Only the piratical disruptions of protesters aboard the Sea Shepherd, who dogged the Japanese whaling fleet earlier this year, prevented the hunting of much-loved humpback whales in 2008.

And it's not only the Japanese. Throughout the world, whales and dolphins – now known to be as intelligent as primates – are still kept as performing animals in oceanariums. Since the 1960s, when killer whales were first taken from the wild, 200 have died in captivity. Whales and dolphins still die, too, in their thousands as "bycatch" in commercial fishing. They suffer from ever louder military and industrial sonar, causing mass strandings in which hundreds of whales and dolphins have been known to perish on beaches, slowly crushing their internal organs under their own weight. Meanwhile, the environmental threat to whales is greater than ever. As oceans warm and the oxygen content of the sea is impaired and acidifies, the zooplankton and small fish on which whales feed move further north, threatening long-held feeding and migration patterns.

Download the new Independent Premium app Sharing the full story, not just the headlines

One hundred and fifty years ago, in the most famous whaling book of all, Moby-Dick, Herman Melville asked presciently, "Does the Whale's Magnitude Diminish? – Will He Perish?" The answer is still hard to pin down.

For the past four years, I have embarked on my own voyage of discovery in the footsteps of Melville's ambiguous narrator, Ishmael. Writing a book about the vexed relationship between man and whale, and making a new BBC documentary, Arena: The Hunt for Moby-Dick, to be shown this evening, has been an eye-opening process.

In the ongoing battle between human history and natural history, one particular species, the sperm whale – Physeter macrocephalus, the real-life model for Moby-Dick – stands alone, not least because its persecution coincides with the rise of the Industrial Revolution. This was a whale seemingly put on earth to advance man's progress. Bizarrely blunt in shape, its pugnacious forehead full of oil, which when pierced, spurted out, causing early sailors to mistake it for the animal's semen – hence its name. But this oil – spermaceti – also supplied the world with light and lubrication. It made fortunes, and enabled empires.

For centuries man's contact with the sperm whale was wreathed in myth and legend. Sperm whales washed up on the coast of Europe were regarded as ill omens, auguring plague or famine. It wasn't until the beginning of the 18th century that they were actively hunted, when the Quaker captain, Christopher Hussey, was out hunting right whales off Nantucket. Blown off course, his ship happened upon a school of sperm whales feeding in the deep waters of the continental shelf. Soon the ports of New England had become the worldwide centre of whaling expertise.

Until the discovery of crude oil in Titusville, Pennsylvannia, in 1859, the world was lit by whale oil. The "Whaling City" of New Bedford, Massachusetts, from which Melville sailed on his 1840 whaling voyage, was the richest city in America. Its ships travelled further and further afield – as far as the South Seas – in search of their precious commodity. They were the equivalent of modern-day oil tankers. They exported the budding economic might of America around the globe – though as Hal Whitehead, the pre-eminent scientist specialising in the study of sperm whales, remarks, the whalers also "left behind diseases, non-native animals (especially rats), technology, and their genes".

Not to be outdone, Britain sent out its own fleet. Whale ships that would otherwise be empty took supplies to Australia. In his 1839 book, The Natural History of the Sperm Whale, Thomas Beale wrote: "Evidence inclines us to believe that these colonies would never have existed had it not been for whaling vessels approaching their shores ... It is a fact, that the original settlers at Botany Bay were more than once saved from starvation by the timely arrival of some whaling vessels." And if they helped lay the foundations of the British Empire, they also helped the country survive two World Wars: in both conflicts, the British fleet harvested whales to turn into nitro-glycerine and thus fuel the fighting machine. Soldiers even treated trench-foot by rubbing whale oil into their feet.

But just as whales helped further the Industrial Revolution, so it hastened their destruction. With the invention of steam engines and grenade harpoons, late-19th- and early-20th-century whalers could pursue the faster rorqual whales – such as the blue whale, the largest animal ever to have lived – which had previously eluded them. In 1951 alone, more whales were killed in one year by the British, Norwegian, Japanese and Soviet fleets than in a century and a half of American whaling.

And as time passed, new uses were found for the arcane commodity of whale oil in the post-modern world. Since it doesn't freeze in sub-zero temperatures, spermaceti was used in Nasa's space missions – no substitute could be found for this natural lubricant. Even now, the Hubble space telescope and the Voyager space probe are careening into infinity, oiled by whales.

Hal Whitehead, the Cambridge-educated scientist who now works at Dalhousie University in Newfoundland, has devoted most of his adult life to the study of sperm whales. He has calculated that their population was reduced from one-and-a-half million before hunting began to a present figure of 360,000.

The whalers often took the biggest animals – the largest bull sperm whales, or their female equivalents – and their removal has had a drastic effect on the current population. Whitehead has proved that sperm whales are highly intelligent animals, capable of communicating in complex series of sonar clicks. The brain of a mature sperm whale weighs up to 17 pounds – the heaviest of any known animal – with a complex neocortex structure. If allowances are made for the animals' blubber, the body-to-brain-size ratio (the Encephalization Quotient, a rough estimate of possible intelligence) of sperm whales indicates significant acumen. Having swum eye-to-eye with these creatures in the Azores – and felt my ribcage scanned by the animals' echo location – I can bear personal testament to their placid and obviously sentient nature.

Studies show that cetaceans can solve problems and use tools, exhibit joy and grief, and live in complex societies. Not only that, but they also pass on these abilities in "cultural transmission" – a gift that has become a problem in itself. Twentieth-century whaling may have destroyed "not just numerous individuals," says Whitehead, "but also the cultural knowledge that they harboured relating to how to exploit certain habitats and areas". The remaining animals have also experienced lower birth rates as a result, and the slow-breeding sperm whale population is growing at a mere 1 per cent a year. The 1986 moratorium, which took effect the following year, may have come only just in time for Physeter.

The blue whale, too, may be recovering from the appalling culls of the 20th century. By the time of the moratorium, the blue whale (Balaenoptera musculus – a little joke by Carl Linneaus, musculus meaning both "muscular" and "mouselike") was regarded as "commercially extinct". In fact, it was commerce, or the lack of it, which ultimately saved it. The hunting of whales stopped largely because there were so few of them left – and cynics note that the move to ban international whaling was championed by America only because its own whaling industry had since fallen into decrepitude. As with the 19th-century abolition of slavery, it was exigency rather than outrage which provided the whale's reprieve.

Neither pro- nor anti-whaling members of the International Whaling Commission ever forget that the 1986 ban was – and is – a temporary and voluntary measure. Norway merely ignored it and continued to hunt minke whales – the same animals watched by its tourists, with disastrous results. One boatload of horrified whale-watchers looked on aghast as the whales they were watching were harpooned by hunters.

But just as the Norwegians claim historical precedence for their whaling, we should also see Japan's actions in context. Their industry is relatively recent (although the island nation has a long tradition of shore whaling). It began in the 1930s, using techniques taught by Norwegian whalers. But it wasn't until the post-war years that it burgeoned, encouraged by the occupying powers under General McArthur, who had the decommissioned ships of the Japanese navy converted into a whaling fleet. One can understand Japanese resentment at what they see as Western hypocrisy.

In a country where whale meat was served in school lunches until the 1970s, it irks to be lectured on the subject. "It's not because Japanese want to eat whale meat," Ayako Okubo, a spokesman for a Japanese whaling company, told the New York Times last year. "It's because they don't like being told not to eat it by foreigners." Some contest that it was America's over-use of pressure on the Japanese – and the moral weight of the environmental lobby – that pushed Japan into its intransigent position.

Indeed, in a contrarian article for Atlantic Monthly, the academics William Aron, William Burke and Milton Freeman argued that "the ongoing campaign to ban all commercial whaling is driven by politics rather than science, and is setting a terrible precedent". The authors proposed that the "cynical actions of the IWC's anti-whaling nations constitute a clear warning to all nations engaged in negotiating multilateral environments". In other words, by saving the whale, the world itself might be lost. Now, as pro-whaling nations push harder for a resumption of legitimate whaling, more pragmatic campaigners say that the only way to stop the Japanese plundering the Southern Ocean is to permit limited commercial whaling, which would allow some control of a currently anarchic situation.

Yet given such pragmatic arguments, after my own experiences I'd still find it impossible to stand on a prow from which an explosive harpoon was about to be fired and not physically restrain the harpooneer from his task. There is no more emotive target in the animal kingdom, and over years of contact with whales, I've been torn between the rational appeal of scientific study and the pitfalls of seductive anthropomorphism.

For the scientists of the Provincetown Center for Coastal Studies in Cape Cod, it is tiny organisms that hold their attention, even though the subject of their research is the North Atlantic right whale (Eubalaena glacialis, "true whale of ice"). Their common name came about because their foot-thick blubber meant that they floated conveniently when dead (rather than sinking to the bottom as other whales will if not first secured). The legacy of hunting has reduced their number to just 350 animals.

Each late winter and spring, these ponderous beasts plough across Cape Cod Bay, hoovering up a ton or more of zooplankton a day to support their enormous bulk. They are a bizarre sight, huge rotund animals surmounted with barnacle-encrusted crests called "callosities". Their baleen – much prized as whalebone in a pre-plastic age, and once used for everything from umbrellas and corset stays to carriage suspension and Venetian blinds – grows in plates up to nine-feet long from their upper jaws. When their garage-sized mouths open, the effect is of some surreal musical instrument.

The right whales choose one of the world's busiest shipping lanes in which to feed, a habit which exposes them, like the humpbacks which share their feeding grounds, to present danger. They may be caught in fishing gear – the Center for Coastal Studies reports that 50 per cent of right whales and humpbacks are thus entangled. Many never escape; others are scarred for life. This summer, I saw one humpback, named Meteor, with a tail so badly torn it resembled the dog-eared page of a book.

If right whales are rare and, for the public at least, difficult to see (only the Center's scientists are allowed closer than 500 feet to the right whales), then humpbacks are a different matter. Prone to surface display, the humpback was called "the most gamesome and light-hearted of all the whales" by Melville, "making more gay foam and white water" than any other. Megaptera novaeangliae ("big-winged New Englander") – its scientific name – is a real trouper.

In my own work with the Center, I've helped identify humpbacks from the black and white patterns on the underside of their tails, or flukes. These patterns are as unique as a fingerprint is to a human, and allow data to be collected about the animal's gender, breeding and patterns of migration. It's a kind of whale census. But the sublime sight of a whale's tail, water running off it like a curtain of mercury before it dips into the sea, is as nothing compared to other humpback behaviour.

A 50-ton and 50ft-long humpback will throw itself entirely out of the water in an act of acrobatics known as "breaching". No one knows why whales do this – and most of the 85 species do. It may be a means of communication; a way of dislodging parasites; or perhaps it's just fun. It certainly looks like it – if I could launch myself out of the water and 20 feet into the air, I'm sure I'd find it addictive too. But what has surprised me, after years of observing these animals, is how often they chose to breach within sight of the whalewatch boats. It is as if they like an audience.

In the evolving relationship between humans and whales, the whale has learned to use man and his machines. Recently, I was on a whalewatch boat with Dennis Minsky, one of the Center for Coastal Studies' naturalists, when a large female humpback came so near that Minsky quipped it could only get closer "if she got into the boat". We could see every detail of the animal. Rolling on its back, the whale displayed the rorqual pleats which expand as it swallows swimming-pool-size gulps of water, then contract to expel the mouthful, catching fish in its baleen like pasta in a strainer. Then, with the nonchalance of a cow in an English field, the female began to rub her belly on the prow of the boat. We were being used as a scratching post.

I've often watched humpbacks blow "bubble nets" underwater, creating spiralling circles to corral their prey, then rising in the middle, mouths wide open, to claim their prize. It is clear, from their proximity, that they use the side of the vessel as a buffer to push their catch to the surface. It works both ways, too. Spanish fishermen work with pods of killer whales which chase their catch into shallow waters, where cetaceans and humans benefit from the fishes' confusion.

Other inter-species relationships are more problematic. In the 1960s, the controversial scientist John C Lilly proposed that whales and dolphins were so intelligent that they should be considered as a parallel, alien life force sharing our Earth. Lilly even declared a new cetacean language, delphinese, and persuaded one female researcher to live in a semi-submerged house so that she could spend more time in intimate observation of her dolphin charges. Unfortunately, Lilly's wilder theories caused him to be shunned by his fellow scientists. Matters weren't helped when he began experiments into the use of LSD on human guinea pigs.

Lilly's left-field claims had the effect of setting back cetacean research – cautious scientists didn't want to be associated with such work. Nowadays, their renewed efforts are concentrated towards ensuring species stay alive. It is another irony that the roll-call of new species is being added to even as recognised species face extinction. In the time it took to write my book, the Yangste river dolphin, a strange, blind, freshwater cetacean, was declared extinct. The North Atlantic right whale will probably go the same way by the end of the century. Yet, amazingly, there are beaked whales or ziphiids – deep-diving cetaceans confined to the open ocean – which have never yet been seen alive, and are known only from bones recovered from beaches. It is a salutary notion that large marine mammals may be swimming in the world's oceans, unidentified by man.

In recent years, the humpbacks of Cape Cod have been particularly plentiful. Thousands of whales return here from their winter migration to the Caribbean. In these fertile, shallow waters, they feed on vast schools of sand eels. At one point last year I found myself on a boat surrounded by 75 humpbacks. In every direction there were spouts and splashes of whales. It was an Edenic sight; the ocean was alive with animals. Even the experienced naturalists I was with put down their cameras and clipboards and stared in wonder.

Maybe we did save the whale after all. As Richard Sabin, curator of sea mammals at the Natural History Museum, told me recently, there have been reports of blue whales swimming up the Irish Sea, and last month, the humpback was officially removed from the endangered list. But that may be an equivocal victory, reviving fears that whaling nations might once again declare open season on these animals. For all their apparently successful recovery, whales still face plenty of threats from the two-legged, gravity-bound creatures with which they happen to share the earth.

Philip Hoare's documentary, 'Arena: The Hunt for Moby-Dick', is on BBC2 tonight at 10.30pm. Tomorrow BBC4 is hosting Whale Night, an evening of programmes devoted to whales. Philip Hoare's new book, 'Leviathan, or The Whale' (4th Estate, £18.99), is out now. To order your copy at a special price (including p&p) call 08700 798 897