The first time Gavin Maxwell saw a basking shark at close quarters was off the east coast of Skye in 1944. It was late afternoon on a luminous Hebridean day: calm sea, light wind, "the hills turning to deep plum" as the sun set. The great fish, nearly 30 feet from nose to tail-tip, was cruising just below the surface, its mouth open to funnel plankton. Maxwell steered his boat to within a yard of the shark, until he could make out its "whole form clear in the transparent water": the beautiful "python-like markings" on its skin, its "frill-like" gills, the brown barrel of its body. Then he opened up on it with a Breda light machine-gun, firing 30 rounds in a single burst "straight into the huge expanse of the flanks". The shark thrashed and dived. By the time it resurfaced, Maxwell had reloaded. He fired another 30 rounds. Six times the shark dived, six times it resurfaced. Maxwell fired a total of 300 bullets into the "broad white target" of ruptured fat on its side. A crew-member thrust a boat hook through its dorsal fin. It dived, dislodged the boat hook, and Maxwell "did not see the shark again".

Yes, it takes a lot to kill a basking shark – as Maxwell was to be reminded often over the following years. Cetorhinus maximus has a tiny brain and super-tough skin, and its vital organs are well protected by its fleshy body. Simple in its anatomy, robust in its armour, this placid plankton-feeder has survived since the early Oligocene, around 30m years ago. Between 1945 and 1949, Maxwell hunted basking sharks off the west coast of Scotland, and he tried many different methods of slaughter. As well as deck-mounted machine-guns, he used shotguns "at point-blank range", aiming "between the eyes" in order to "completely obliterate" the brain. He experimented with harpoon designs. Once, having successfully harpooned a big shark, he tried to drown it by dragging it alongside his boat with its head astern, so that the back-pressure of the water forced the gills open and stopped it breathing. No luck: "It was a surprise to see that after several miles it was still very much alive." When he reached the pier at Mallaig, Maxwell arranged for the fish to be winched aloft and then swung onshore. But as its head emerged from the water there was "a tremendous crack and a sickening tearing noise": the shark's body ripped free from its tail and plunged to the bottom of the harbour, where it bled to death overnight.

Like most people, I was a child when I first discovered Maxwell's writing, reading a well-thumbed copy of Ring of Bright Water (1960) and weeping at the death of Mijbil the otter. "Camusfearna" and "Sandaig" were place-names to be conjured with in our family, serving as shorthand for a Scottish west-coast wildness. Maxwell seemed to me then a visionary saint, austere in his hideaway, capable of magical communion with creatures.

All that changed when I read Harpoon at a Venture (1952). It is an extraordinary work: a militarised Moby-Dick, an epic of abjection, a saga of cetacide and capital. In it Maxwell – Queequeg with an Oerlikon, Ahab with an inheritance – demonstrates an awesome lack of interspecies empathy, and a broader insensitivity to affect that verges on the psychopathic. The book, Maxwell's first, details his attempt to establish a commercial basking-shark fishery on the tiny Inner Hebridean island of Soay in the years after the second world war. Born into minor aristocracy, Maxwell used family money to buy Soay, with the hope of turning it into his "Island Valley of Avalon" – a place to escape the crush and death of conflict.

Except that his Soay "adventure" turns out to be a continuation of war by other means. The Hebridean seas become a new kill-zone, in which Maxwell can surround himself with men and weaponry and resist the "ennui of peacetime". Harpooning is "bayoneting". Waves crash on headlands with "a boom like distant artillery". Shooting rare white-sided dolphins as they leap is a kind of anti-aircraft fire. When the fishery gets going, the shore of Soay is "a tremendous and terrible sight", with corpses lying in "long rows at the tide-line, the blood trickling down among the stones, and the sea behind them … crimson for hundreds of yards, a true sea of blood". It is impossible not to think of the Normandy beaches during the D-day landings – the planning and training for which Maxwell had been involved as an SOE agent.

Maxwell loads his harpoon gun off the coast of Soay. Photograph: Raymond Kleboe/Getty Images

War by other means, then – and sex also. Harpoon trembles at times under the strain of suppressing Maxwell's then-illegal homosexuality. The book opens with him entering a "communal shower-bath" in 1940, after a night working in a mobile anti-parachute unit during the Blitz: "I elbowed my way between two naked guardsmen, one of whom stood ludicrously to attention." This upstanding soldier turns out to be Hebridean, and his "soft speech" prompts in Maxwell a "momentary vision" of a Scottish island "against a harsh Atlantic sunset": the dream that will be realised as Soay. The fishing itself is rampantly sexualised: Maxwell writes of the "viscous slime" of the shark's skin, the "penetration depths" of the harpoons, the "drench of white spray" that accompanies harpooning, and "the shuddering wrench" and "impotent effort" with which the sharks thrash after being wounded.

In the years Maxwell hunted them, basking sharks were classed as vermin for the damage they did to the nets of herring fishermen. There was a small bounty on their heads, but the real money lay in their livers, which held gallons of lucrative oil. The difficulty was in the extraction of that oil – and much of the book is preoccupied with this pragmatic puzzle: how to locate the sharks, harpoon them, transport them back to the processing plant at Soay, land them, eviscerate them, remove the profitable parts and dispose of the remainder. None of this was easy with a creature that can grow to 40 feet in length and 20 tonnes in weight.

So much goes wrong. The harpoons buckle, the guns fail, the sharks escape and a rival company moves into Maxwell's fishing grounds. There is feuding among the crew and drunken discontent at the processing plant. A freak cyclone flattens the buildings on Soay. Maxwell is almost knifed. His right-hand man – the "indestructible" Tex Geddes – is nearly drowned and then nearly crushed. Red tape entangles the enterprise. Bankruptcy constantly threatens. "No sooner was one difficulty overcome than another appeared," writes Maxwell, wearily.

The beauty of the Hebridean coast thrills Maxwell, and seabirds hold a "fascination" for him, but he is oddly incurious as to the sharks themselves: he might as well be hunting sofas. There are, it is true, passages where he evokes the strangeness of their existence: the sperm that flows as "hundreds of semi-opaque milky globules like golf-balls, varying in size, and looking as though made of Lalique glass". Once he sails over a huge school of shark and gazes down on them, "layer upon layer, huge grey shapes like a herd of submerged elephants, the furthest down dim and indistinct in the sea's dusk". But the book's prose really sings when he is writing of the land, light and water of the Hebridean seaboard – or when action occurs. In one astonishing episode, they harpoon a shark and are still fighting it when night falls, a gale blows up and the sea all around them blazes with phosphorescence. "We sailed a dream sea in the dark and the eerie phosphorescence," writes Maxwell, "towed by the wounded shark far below us in the dark water".

Soay itself soon becomes an abattoir: the Hebrides reimagined by Bosch and Goya. The skin of basking sharks is spined, rasping and parasite-pitted: Maxwell and his fellow butchers wear "armoured gloves", and wield "axes, saws and knives". He spends days at sea hunting, and days on Soay struggling in "mountains of soft cold flesh and entrails". To get at the liver, it is necessary to cut deeply and quickly across the belly, so that an "avalanche of liver and entrails" rushes out. "Once I was not quick enough," writes Maxwell, "and was knocked flat on myback and enveloped by it, struggling free drenched in oil and blood, with a feeling almost of horror." Of that grotesque sentence, it is the word "almost" that is most disturbing.

How far all this is from today's ethically well-intentioned nature writing. How far, too, from the widespread perception of Maxwell as a man who lived in harmony with the wild world. Harpoon is about blood and bone and blades and ledger-books; about how chunks of shark flesh continue to quiver eerily for hours after death, even if the "entire fore-part of the head" has been severed with a hatchet.

Which is what makes it, and pretty much everything Maxwell wrote, so fascinating. His books represent – in their psychodramas and their ultraviolence – the dark side of British place-literature. To read them as hymns to tranquillity is trite. To engage with their tangled understories is mesmerising. Alongside them I would place TH White's The Goshawk and JA Baker's The Peregrine, which reads – in its obsessive tallying of body parts, bloodstains and kill paths – like an ornithological CSI.

By the summer of 1947, Maxwell had refined his shark-hunting techniques. He and his men were killing fish in such numbers that they overwhelmed the processing system on Soay. The water in the little harbour was rank with bloodied scum, and writhed through by huge conger eels. The concrete was heaped with "gigantic piles of offal". Pink plankton spilt from the slit stomachs of the sharks gave a lipstick sluice to the slipway. Thousands of gobbet-glutted gulls lazed on roof-ridge and boulder. Pied wagtails hunted flies among the vertebrae strewn over the field behind the factory.

The centre of this putrescence was the factory pickle-tank, in which had been stowed 16 tonnes of shark flesh. But the pickling solution was of insufficient strength, and the flesh had begun to rot. By the spring of 1948 it had become a "nightmare cave", and when Maxwell lifted its hatch the smell, "ammonia, dense, suffocating and almost visible", knocked him back "as completely as a robot fist". He steeled himself to look down into it again. The surface of the contents seemed to be seething like "an obscene sea". It was, he realised, "a million million grubs", heaving on the fish-flesh. Avalon had been consumed by Beelzebub. In July, Maxwell formally resigned as director of the Island of Soay Shark Fisheries Ltd, and soon after he sold up and left Soay.

Eight years later, he retreated to Sandaig, looking across the Sound of Sleat to Ornsay, where he had machine-gunned that first shark. It was at Sandaig that he would write Ring of Bright Water, the book that would inspire countless conservationists. A stone plaque there now marks the grave of Edal, the otter who died in the fire that also destroyed the house. Cut into the stone is one of Maxwell's most-quoted edicts: "Whatever joy she gave to you, give back to Nature."