Provincetown, a port that once serviced the whaling industry of New England, sits at the tip of Cape Cod on the eastern seaboard of the United States. The philosopher Henry David Thoreau called it a place where "a man might stand and put all America behind him". But its history is not exactly peaceful. In the 19th century, hundreds of ships left here to slaughter thousands of whales around the globe.

Now, ironically, it is one of the best places in the world from which to go whalewatching - a vivid reminder of how quickly we have moved from seeing whales as a resource to be consumed to a natural wonder to be conserved.

I first came to Provincetown in 2001, but it was only on the last day of my visit that I took a whalewatch trip. As a boy I'd been fascinated by whales, but that fascination had lain dormant for years. It was about to be powerfully re-invoked.

As we sailed out of Provincetown harbour, we left the calm protected waters, passing breakwaters colonised by cormorants and lounging seals. A series of lighthouses, icons of Cape Cod, signalled that we were entering the open ocean.

I don't know what I expected to see that day. My only previous experience of living cetaceans (the collective name for whales and dolphins is cetacea, Greek for sea monsters) was of captive animals, a distinctly depressing sight. A tame orca in an overgrown swimming pool in Windsor safari park; performing dolphins in Brighton's underground car park of an aquarium; a pair of blemished and pathetic beluga in a tank on Coney Island. Guilt dominated those visits. What I saw in the Atlantic that afternoon overturned my pessimism about humanity's interaction with whales.

In the distance, we saw seabirds circling - a good indication of whales ahead. The next came from the animals themselves: distant blows, breaking the horizon like individual clouds, as if their creators were carrying their own weather with them.

Experts can identify cetacean species merely from the shape of such airy semaphores (a whale's blow is not sea water but condensed breath, much as you see your breath on a frosty morning). A fin whale shoots a tall column in the air; a right whale sends a v-shaped spout from its twin nostrils; a sperm whale's blow is sharply cantered to the left, from a single nostril.

Such signs were once advertisements to whale hunters. Now they announce an imminent miracle: the bushy blows we could see were those of humpback whales.

One of nature's greatest displays

In Moby-Dick, Herman Melville calls humpbacks "the most gamesome and light-hearted of all the whales". Other whales betray little of themselves at the surface. Lay in wait for a minke - one of the smaller baleen whales (even though an adult is bigger than a London bus), and the most you are likely to see is a sharp-pointed snout (hence its Latin name, Balaenoptera acutorostrata), followed by a disappearing back.

Equally, the fin back, Balaenoptera physalus - at a maximum of 26m (85ft), second only to the blue whale in size - is, for all its vastness, just as elusive, allowing a glimpse of barely a tenth of its body. You must simply imagine, as this great creature speeds by at 20 knots or more - earning its nickname, the greyhound of the sea - the vast, streamlined torpedo that lies below.

The humpback is a different prospect. Its Latin name, Megaptera novaeangliae, translates as "big-winged New Englander". And in a few moments, I was to discover exactly why.

As we came upon the feeding whales, surmounted by flurries of opportunistic gulls, we saw distinctive grey-green patches rising through the water. For any whalewatcher, this is the most immediate intimation. In a unique technique, humpbacks blow precisely calibrated streams of bubbles to create underwater nets around their prey - tiny sand-eels. In these fertile waters, schools of millions of sand-eels can stretch over areas the size of playing fields. They provide the main source of food for humpbacks; an adult whale will eat a tonne of fish a day to sustain its vast bulk.

As I watched the spiralling bubbles rise to the surface, the excitement was intense. Even our naturalist's voice went up an octave as he told us what we were about to see.

Suddenly, the surface broke with the yawning, gaping mouth of a monster. The whale had appeared right beside the boat, gulping hundreds of gallons of seawater.

I was peering into the jaws of the leviathan.

I could see sand-eels leaping out of the animal's mouth in a vain attempt to escape. Using its baleen - the fibrous plates that line its upper jaw instead of teeth - the whale was catching fish like pasta in a strainer. I could even smell its breath - pretty bad, if you're wondering - and as it dived again to repeat the process, the spray from its blow hit my face. It felt like a kind of baptism.

Watching humpbacks bubble-feed is one of nature's greatest displays. I have seen it many times now, but it never fails to amaze.

On one research trip with Dennis Minsky from the Provincetown Centre for Coastal Studies, we were surrounded by whales in every direction, more than 100 humpbacks feeding en masse. Minsky just put down his camera and clipboard and gaped in astonishment. The forest of blows and their elephantine trumpets seemed to celebrate a lost Eden; a vision of a time before man had made his irrevocable mark on their world.

But humpbacks have something even more spectacular in their repertoire.

Breaching is another of the whalish mysteries, a phenomenon observed by the ancient Greeks, yet unexplained by modern scientists. And nothing prepares you for the sight. That first trip, as I stood at the prow, without warning a 50ft, 50-tonne humpback launched itself entirely out of the water.

For a split second the animal appeared like some vast and improbable whale-angel against the sky, its huge, gnarled flippers outstretched like wings. Every detail was visible. I saw its great ribbed belly, the rorqual pleats that expand when feeding. I saw the barnacles on its skin, the parasites that hold fast to the animal, making it a travelling colony of its own. Then, as if someone had taken their finger off the pause button, the animal bowed to gravity and fell back into the sea, creating a splash that resounded for miles.

Forgetting that I was surrounded by schoolchildren, I blurted out an inadvertent, "Fuck!" Hardly an erudite response, but I challenge anyone to be indifferent to a close encounter with a whale. I have seen grown men cry at their first sight of a cetacean. They simply exist in another universe; aliens occupying vast oceans of which we have less knowledge than we do of the surface of the moon. To see a whale is a privilege. But it can also become an obsession. This spring, I succeeded in a long-held ambition: to watch right whales from the shore.

Each year, these rotund, blubbery whales - so-called because they were the right whales to hunt, since they floated when dead - assemble in Cape Cod Bay. The North Atlantic right whale (Eubalaena glacialis) is one of the most endangered animals on earth; fewer than 400 individuals remain. Yet if you are lucky, you can see these magical creatures just tens of feet from the cape's outer shores.

The most mysterious of all whales

As I trekked out to a remote strand, a curtain of gulls rose to reveal a group of whales so close I could almost have waded out to them. A glorious welter of black fins and flukes, they were interacting with paradoxical sensitivity, ignorant of my presence, or of the sole harbour seal that stayed at the edge of the surf, as if reluctant to share the waves with these cavorting behemoths. I almost felt I was intruding.

My experiences in Cape Cod were extraordinary, but as a confirmed whalehead, I was now eager to see other species. In the summer of 2007, along with the crew of the BBC Arena film, The Hunt for Moby-Dick, I travelled to the Azores. There I came closer to whales than I have ever done before.

One thousand miles due west of Lisbon, the Azorean archipelago lies in the middle of the Atlantic, caught between three tectonic plates and surrounded by some of the deepest seas in the world. Only 100m out, its volcanic shores plunge to a mile in depth - then descend even deeper to the abyssal plain. These waters are a perfect environment for the most mysterious of all whales.

Sperm whales were prized by hunters for their oil; it burned clear and smokeless in lamps and made pure white candles. For more than a century, sperm whales lit and lubricated the industrial revolution. As a result, their numbers were drastically reduced from three million to a tenth of that number; a toll exacerbated by the mechanised hunting carried on in the 20th century by Britain, Norway, Japan and the former Soviet Union. Even the Azores had their own whale-hunt, using methods learned from the Yankee whalers, which ended only in 1986. Ominous stone slopes still mark whaling factories were whales were hauled out of the depths and ground down into fertilizer.

Now islands such as Faial and Pico have turned to watching, rather than hunting. Zooming out of Lajes harbour on a rib (rigid inflatable boat), we sped into the deep blue Atlantic, common dolphin riding our bow like competitive racers, gloriously streamlined. They were a good sign. Ahead, we saw blows - but utterly unlike any I'd ever seen before. Distinctly right-angled, they came from what looked like logs lying in the water. As João, our captain, cut the engine and the boat bobbed to a halt, the shapes resolved themselves into animate objects.

Grey heads rose out of the waves, blunt and almost cartoon-like. It was difficult to tell which end of the animals was which until their massive flukes drew up and dived in a choreographed departure. For such gigantic creatures, they left no whirling vortex behind, only a calm circle of spinning water. The Inuit believe these qaala, as they call them, to be mirrors into the whale's soul; and mirrors into ours.

As the whales dived, our first mate, Marco, lowered a hydrophone over the side. We listened to the clicks of the animals hunting squid, hundreds of feet below. As we waited for the whales to reappear, alerted by their eerie morse code, I felt a certain anxiety. The Azorean government had given me permission to get into the water with the whales - something I had dreamed of all my life. But when the moment came, there was barely any time to think about what I was doing. As we approached a school of 12 sperm whales, João urged me into the water. I jammed on my fins and pulled on my snorkel and mask.

Visibility was about 20ft ahead as I swam, unseeing, towards the whales. Suddenly they were there, filling the entirety of my vision, lying peacefully in the shafts of sunlight that pierced the water. Then, as if in slow-motion, one large animal detached itself from the group, and moved towards me.

For seconds that seemed like hours, I thought its great granite head was about to collide with me - or, perhaps, open its jaws and swallow me whole.

Sperm whales are the only cetaceans which could swallow human beings, and have done so. Instead I felt, rather than heard, the animal's echolocation focusing on my puny body. Its sonar was scanning my skeleton like an MRI, recreating an image of me in its head, diagnosing me inedible.

I think it is the silence of the encounter that stays with me. That, and the utter placidity of the huge animal. Silhouetted against the blue, the whale turned and looked at me, eye to eye. It was the most disconcerting moment of my life. Then it dived, perpendicularly into the profound blackness, and was gone. That night, I couldn't close my eyes. Every time I did, the whale swam into my head. It has yet to leave my dreams.

• Philip Hoare's Leviathan or, The Whale, winner of the 2009 BBC Samuel Johnson Prize for non-fiction, is published by Fourth Estate, £8.99. Arena: The Hunt for Moby-Dick will be repeated on BBC2 later this month.