We love - and hate - our neighbours because our genes make us do so. And there's algebra to prove it. Andrew Brown tells the strange story of the men who unravelled the maths in our human make-up: knowledge that, for one of them, was too much to bear

The deathbed of an altruist can be a terrible place: "A mattress on the floor, one chair, a table, and several ammunition boxes made the only furniture. Of all the books and furnishings that I remembered from our first meeting in his fairly luxurious flat near Oxford Circus there remained some cheap clothes, a two-volume copy of Proust, and his typewriter. A cheap suitcase and some cardboard boxes contained most of his papers, others were scattered about on ammunition chests."

These were the effects of George Price, an American science journalist. He had perfected an existing mathematical equation that shows how altruism can prosper among basically selfish animals - even humans. So shocked was he by his success in this, and the darker truths about human nature implied by the equation, that he embarked on a desperate career of service to the outcast, and finally killed himself with a pair of nail scissors in a London squat in January 1975.

Download the new Independent Premium app Sharing the full story, not just the headlines

The equations for altruism are not a figure of speech. They were first discovered by WD Hamilton, an Oxford biologist, who tells the story in his newly published collected papers. He is now Royal Society Research Professor of Zoology at Oxford, laden with scientific honours, but was then scrabbling around the fringes of the academic world with only a second- class degree from Cambridge, poor and so lonely that he sometimes worked at night on a bench in Waterloo station rather than return to his bedsit.

Even now, when he is one of the most revered biologists in the world, there is an extraordinary shyness and simplicity about his manner. When I went to see him in his office, he had entirely forgotten our appointment, and yet talked for 40 minutes with the utmost courtesy.

The first, clumsy equations that he produced represent one of the great explanatory triumphs of Darwinism. They show how genes for self-sacrificing behaviour can spread through a population even though they harm some carriers of the genes in question. They demonstrate how animals can develop astonishingly selfless behaviour: how bees can evolve that sting fearlessly even though they must die in consequence. The secret is to ensure that altruistic actions also benefit relatives of the altruist - who are themselves likely to share the gene in question. This helps to clarify why a mother may lay down her life for her children. But how much should she risk for a third cousin twice removed? The equations produce answers to such questions for every living thing on earth.

When Price first read Hamilton's equations he recognised that they raised a terrible problem. He saw that altruism in this biological and equation- bound sense is limited. It cannot supply the absolute and universal commandment of Christianity or the other global ethical systems.

The Hamilton/Price equations may tell us we must love our neighbours, but in ways that are about as far from the religious sense of the words as possible. They are descriptive, not prescriptive. There is no "ought" about their command to love. We love our neighbours because our genes built us that way, the equations say; and because the neighbours have probably been built the same way, too, and so will love us back.

This insight so shocked Price that he set out to check Hamilton's work himself and find the flaw he was convinced must be there. Instead, he ended up with a more elegant and general way to express them.

This new formulation made even clearer a worrying implication that he had already grasped: that the same equations that demands the spread of altruistic behaviour may sometimes demand its opposite. He recognised that a fondness for torturing, raping and murdering your neighbours is just as heritable and may be as easily spread as the urge to love them.

When Price discovered this, he was a militant atheist. Indeed, his atheism had played a role in his divorce from a Catholic in America and emigration to London. The discovery plunged him into a severe depression, from which he was delivered by an experience of God. "He never described it to me in detail," says Hamilton, explaining the story in his recent book. "He could tell that I was practically as he had been in his former life, and not open to anything that would seem intrinsically supernatural.

"He described himself running through the streets of London in the neighbourhood of his flat in Marylebone. He was looking for a church, he said, and entered the first he came to and prayed for guidance. The immediate result was his complete dedication to Christianity."

As an atheist and materialist, Price had been insufferable: for instance, he had proposed controls of almost impossible stringency on any experiment designed to prove ESP to eliminate the remotest possibility of fraud. As a Christian, he was just the same. He soon quarelled with with the priest who received him, whom he found insufficiently zealous. He was not a fundamentalist in any normal sense: he completely accepted Darwinian evolution, and continued to work on his equations. He did not believe in the literal truth of biblical narratives. But he seems to have heard the sayings of Jesus as directly and unarguably as a bee feels the imperative to defend its nest.

"Sell all you have and give to the poor." The derelicts he entertained stole from him and caused chaos. He was forced to leave his apartments, and ended up dossing on the floor of the lab at University College, London, where he worked. Not even that lasted. An alcoholic whose wife he had tried to help started to harass him at the lab, and finally took to shouting at him from the street below. So he had to leave there, too, and descended by degrees to the squat in Tolmers Square, Euston, where he killed himself. His funeral was attended only by three or four tramps, and two of the most honoured and influential biologists in Britain, John Maynard Smith and WD Hamilton.

The story emerges from the prefaces to Hamilton's collected papers, which have just been republished (Narrow Roads of Gene Land by WD Hamilton, published by WH Freeman /Spektrum). The papers themselves are dense, real science. The innumerate reader must force himself past thickets of equations in search of the enchanted castle of truth - only to find that the equations were the truth. The prefaces to each paper, however, which describe how and why they came to be written, are a delight for anyone who is interested in the world, or even in English prose.

What is it about biologists that makes them write so well? Richard Dawkins is the obvious example, but not the only one: Stephen Jay Gould, and EO Wilson, who coined the term sociobiology, would serve almost as well. Hamilton has spent his working life in the study of two great literary motifs, sex and altruism, and as his description of George Price's deathbed, quoted at the beginning of this essay, shows, he has most of the gifts a novelist needs.

Perhaps this vividness arises because biologists believe they have a clear and important message about human beings and our place in the world. The Hamilton/Price equations show how many of the traits that we think of as peculiarly human can arise and spread among animals. They suggest there must be biological limits to the general level of altruism in any human population, no matter now much or how little love may be shown among the extremes. They seem to annihilate the special status of human beings in the world. They seem also to render religious accounts of human nature unnecessary. Dawkins's brusquely unsympathetic view of religion is well known. Wilson, founder of sociobiology, is just as hostile to the truth- claims of religion, though far more sympathetic to their emotional attractions. Hamilton himself, in conversation, says that Price, before his conversion, was "much more of an atheist than I am".

The equations for altruism are only a part of the sociobiologists's armoury. By treating behaviour patterns as inherited, and as much subject to Darwinian evolution and susceptible to mathematical analysis as are the body shapes of animals, they have discovered a set of tools that can appear fearsomely powerful. Hamilton's equations appear to predict, for example, that if our immediately ancestral chimp species did make its crucial transition towards humanity on a sea shore, then we are likely to be more co-operative and altruistic by nature than if this happened in a jungle or savannah. This is because with movement limited to one long strip of coastline, neighbouring groups are more likely to share genes than if there is a whole savannah through which to disperse.

But an account of how human morality might have evolved does not solve moral questions. Knowing that a conscience is part of our nature does not tell us how we should keep on good terms with one. Evolutionary theory is a science of averages and mathematical abstractions, whereas we live our lives uniquely and unrepeatably. The altruistic or selfless acts that interest us as ethical beings are those that people choose freely, not those we cannot avoid: the man who dies in the cellars of the secret police rather than implicate his friends in a conspiracy is a heroic figure, whereas the baby who dies in one of Ceausescu's orphanages from emotional neglect is merely tragic.

The same complexities arise if we consider selfishness instead of altruism. The Selfish Gene, Richard Dawkins's first book, was partly a popularisation of Hamilton's work and its implications. And genes for altruism are "selfish" to exactly the extent that all other genes are. They compete to pass themselves on into subsequent generations, and if we have them, we know that they have competed successfully. But this "selfishness" is an abstract quality. It is nothing like the selfishness we talk about when discussing people's character. The links between genes and human behaviour are found at a much darker and more atavistic level: one of the most chilling moments in Hamilton's book comes when he reflects on his own dark, violent passions so alien to his civilised self, and concludes that they must represent a substrate of primeval nature: one of the things that let his ancestors survive.

George Price's life and death does not simply illustrate the extraordinary complexity of the relationship between religion and science. It also shows the vast gulf fixed between what a biologist might mean by selfishness and what an novelist might. The last act of Price's life was in one sense supremely selfish, as suicide often is. But even that had its considerate aspect, as Hamilton tells it: "As I tidied up what was worth taking into his suitcase, his dried blood crackled on the linoleum under my shoes: a basically tidy man, he had chosen to die on the open floor, not on his bed."