Steven Pinker is one of those wunderkinder that elite US universities seem to specialise in producing. Born in Canada in 1954, he's currently a professor of psychology at Harvard, but ever since he arrived in Cambridge, Massachusetts in 1976 he's been bouncing like a high-IQ tennis ball between Harvard and its prestigious neighbour, MIT (he has professorial chairs at both institutions). By profession he's an experimental psychologist who began doing research on visual cognition but eventually moved into studying language, especially language acquisition in children. He probably knows more about mankind's use of verbs, and particularly the distinction between irregular and regular ones, than any other man, living or dead.

He's also a prolific and high-selling author of popular books about his subject, which has led some unkind academics to describe him as "the thinking man's Malcolm Gladwell", which is unfair because Pinker, unlike Gladwell, is a distinguished scientific researcher; in fact the only characteristics he and the New Yorker writer have in common are big hair and an enviable fluency that translates into publication track records as long as your arm.

Pinker's assault on the common reader began in 1994 with The Language Instinct, an accessible introduction to the idea that humans are "language animals", biologically wired for linguistic communication. In 1997 he published How the Mind Works, which went beyond language to offer a similar portrayal of the rest of the mind, from vision and reasoning to emotions, humour and art. In 1999 he returned to the question of language with Words and Rules: The Ingredients of Language, which drew on his research on regular and irregular verbs as a way of explaining how language works in general. And in 2002 he published The Blank Slate: The Modern Denial of Human Nature, which is basically an attack on what Pinker sees as three great misconceptions about human behaviour: the idea that the mind is a tabula rasa, a blank slate that is wholly shaped by one's environment; the notion of the "noble savage", the idea that humans are intrinsically good but get warped by society; and "ghost in the machine" theories that postulate the existence of a non-biological agent in the brain that can alter human nature at will.

The Better Angels of Our Nature: The Decline of Violence in History and its Causes (Allen Lane) is Pinker's latest and most ambitious book. The title is taken from a phrase of Abraham Lincoln's. In it, Pinker challenges one of our deepest but unexamined assumptions – that current and recent times have been the most violent in human history. As evidence we cite contemporary homicide rates, the Holocaust, the death toll of two world wars and the genocide adventures orchestrated by Stalin, Mao Zedong and other tyrants. But Pinker argues that this view is radically mistaken. Violence within and between societies – both murder and warfare – has declined from prehistory to today. We are, he maintains, much less likely to die at someone else's hands than ever before. And even the horrific carnage of the last century, when seen in the long view of history, is part of this trend.

This is a big idea if ever I saw one, and it requires a massive tome (700 pages plus footnotes) to deal with it. In the first place, Pinker has to locate, analyse and explain the empirical and other data that support his thesis: that, however you measure it, the past was not just a different country, but also a far more violent one. And then he has to provide some explanations for why the long-term reduction in violence happened. To do that he ranges far beyond his own professional territory – into forensic archaeology, political philosophy, intellectual and social history, population dynamics, statistics and international relations. He identifies a number of forces that were key factors in curbing mankind's capacity for inhumanity: the slow emergence of states capable of playing the role of Hobbes's "Leviathan"; the pacifying impact of commerce and trade on behaviour; the impact of the Enlightenment on the way people thought about others; the evolution of notions of etiquette over the centuries; the way print and literacy expanded the "circle of empathy" beyond people's immediate family; the importance of women in civilising men; and the "long peace" that followed the second world war.

The Better Angels is a long, absorbing and sometimes horrifying book, because in order to establish his case Pinker has to dwell at some length not just on the savagery of the past, but on the way brutality and cruelty was – until relatively recently – taken for granted. If you want to know about medieval forms of torture, or the favourite tools of the Inquisition, or how Tamerlane's troops operated, then you will find ample material here. The ingenuity of human barbarism knows no limits. What's even more salutary, however, is the realisation that it's not all that long ago since people were routinely hung, drawn and quartered in England; or that flogging and keelhauling were routine methods of maintaining discipline in the Royal Navy; or that nobody batted an eye at the flogging of children as late as the 1950s.

In one sense, this is also a consoling book, in that it describes mankind's path from unimaginable barbarism to a safer, more "civilised" state. Critics might describe it as an example of the Whig interpretation of history – that is, an account of inexorable progress. But that's not how I read it. Pinker's account suggests that what the great cultural historian Norbert Elias called the "civilising process" was a long, arduous struggle whose long-term outcome was never assured. And recent increases in violence – such as the surge in homicides in the 1960s in the US, or the chronic violence that is now disfiguring much of Latin America – suggest that it's still not assured. Perhaps the wisest thing we can say about the trajectory described by Pinker is what the guy who jumped from the 50th floor said when he was passing the 25th: so far so good.

As with all his previous books, The Better Angels has been taken seriously by heavyweight reviewers and, broadly speaking, came through with lots of plaudits. Writing in the Guardian, for example, Cambridge political philosopher David Runciman lauded it as a "brilliant, mind-altering book". In his New York Times review, Princeton philosopher Peter Singer described it as "a supremely important book. To have command of so much research, spread across so many different fields, is a masterly achievement. Pinker convincingly demonstrates that there has been a dramatic decline in violence, and he is persuasive about the causes of that decline."

Clive Cookson, the Financial Times's science editor, found some passages repetitive and worried that "readers of a squeamish disposition might feel that he has included too many detailed accounts of murder and excruciating torture through the ages, in his effort to illustrate how deeply brutality was once woven into the fabric of daily existence". But, overall, Cookson thought that The Better Angels was "a marvellous synthesis of science, history and storytelling". In the Wall Street Journal, James Q Wilson found the book "a masterly effort to explain what Mr Pinker regards as one of the biggest changes in human history: we kill one another less frequently than before" but thought that "to give this project its greatest possible effect, he has one more book to write: a briefer account that ties together an argument now presented in 800 pages and that avoids the few topics about which Mr Pinker has not done careful research".

The major voice dissenting from this chorus of praise belonged to political philosopher John Gray, who was unimpressed by Pinker's claim that the Enlightenment was a key factor in civilising humanity. Writing in Prospect magazine, Gray attacks what he sees as Pinker's identification of "the Enlightenment" with a carefully chosen but not necessarily representative group of thinkers. "These are highly disparate thinkers," says Gray, "and it is far from clear that any coherent philosophy could have 'coalesced' from their often incompatible ideas. The difficulty would be magnified if Pinker included Marx, Bakunin and Lenin, who undeniably belong within the extended family of intellectual movements that comprised the Enlightenment, but are left off the list. Like other latter-day partisans of 'Enlightenment values', Pinker prefers to ignore the fact that many Enlightenment thinkers have been doctrinally anti-liberal, while quite a few have favoured the large-scale use of political violence, from the Jacobins who insisted on the necessity of terror during the French revolution, to Engels who welcomed a world war in which the Slavs — 'aborigines in the heart of Europe' — would be wiped out."

Intrigued by these reactions, I contacted Professor Pinker in his Harvard lair and exchanged some emails with him. Here is the log of our conversation.

Steven Pinker and John Naughton: an exchange

JN: What got you started on this huge undertaking? Was it really finding Ted Gurr's study of homicide rates in various English towns between 1200 and 2000?

SP: That little-known decline certainly lodged in my memory when I first read it, together with other observations I had made at various times – that tribal warfare was proportionally deadlier than our world wars; that many barbaric customs, such as slavery and heretic-burning, switched from unexceptional to abhorrent and were never defended again; that the Soviet empire vanished in a puff of smoke with very little violence. But it was only after reiterating these observations in a web forum, and then getting emails from a number of scholars telling me that there was far more evidence for a historical reduction in violence than I had cited, that I realised that something systematic was going on. Various experts told me that every European country, not just England, had enjoyed a homicide decline since the middle ages; that the number of wars and the number of deaths in wars had plummeted in recent decades; even that spousal and child abuse had declined. I realised that someone had to try to tie all these developments together.

Worst atrocities This table takes past conflicts or tyrannies and recalibrates their death tolls so their scale can be compared directly Conflict Century Death toll* Death toll (20C equivalent)** Ranking 1 Second world war 20th 55 million 55M 9 2 Mao Zedong (mostly government-caused famine) 20th 40M 40M 11 3 Mongol conquest 13th 40m 278m 2 4 An Lushan revolt 8th 36m 429m 1 5 Fall of the Ming dynasty 17th 25m 112m 4 6 Taiping rebellion 19th 20m 40M 10 7 Annihilation of the American Indians 15th-19th 20m 92m 7 8 Josef Stalin 20th 20m 20m 15 9 Middle East slave trade 7th-19th 18m 132m 3 10 Atlantic slave trade 15th-19th 18m 83m 8 11 Timur Lenk 14th-15th 17m 100m 6 12 British India (mostly preventable famine) 19th 17m 35m 12 13 First world war 20th 15m 15m 16 14 Russian civil war 20th 9m 9m 20 15 Fall of Rome 3rd-5th 8m 105m 5 16 Congo Free State 19th-20th 8m 12m 18 17 Thirty years' war 17th 7m 32m 13 18 Russia's “time of troubles” 16th-17th 5m 23m 14 19 Napoleonic wars 19th 4m 11m 19 20 Chinese civil war 20th 3m 3m 21 21 French wars of religion 16th 3m 14m 17 *Median/mode of figures cited in encyclopaedias or histories. Includes battlefield and civilian deaths **Deaths were calculated against global population at time, then scaled up to mid-20th century level

JN: George Bernard Shaw was once asked at the publication of one of his books why he'd written such a long tome and famously replied that it was because he wasn't clever enough to write a short one. Having read my way through your 800-odd pages I've come to the conclusion that Shaw got it the wrong way round. Indeed, the thought that came to my mind is that Better Angels feels like the kind of thing that was once produced in Britain by royal commissions – that is to say, massive investigations of big subjects conducted by committees of experts who are supported by small armies of smart civil servants. Yet your book is clearly the product of a single intelligence, so I'm curious to learn how you went about producing it. Did you have a small army of research assistants? And how long did it take?

SP: I had one research assistant, who did the graphing, spreadsheet-crunching and internet surveys. Even with those, I scrutinised the raw numbers to make sure that nothing screwy was going on with my data sources, such as wars being double-counted (for example, the second world war in Europe versus the second world war overall) or some decline being an artefact of a shift in inclusion criteria.

I spent a bit more than a year doing nothing but reading, to educate myself in fields I was not trained in – mainly criminology, history and international relations. The writing took 14 very intensive months. I had a sabbatical from Harvard, moved to our house in Cape Cod, and worked on it day and night seven days a week, taking time out only to exercise and spend time with my wife, novelist and philosopher Rebecca Goldstein.

I depended on guidance and feedback from Rebecca and from experts in many fields, whom I badgered for advice, clarification and feedback, on everything from which data sources to trust, to whether data in one of their graphs went from 1800 to 1824 or from 1801 to 1825.

JN: When you're asked by talk-show hosts to explain in a nutshell what the book is about, what do you say?

SP: Believe it or not, violence has been in decline for long stretches of time, and we may be living in the most peaceful era in our species' existence. The decline has not been steady; it has not brought violence down to zero; and it is not guaranteed to continue. But it is a persistent historical development, visible on scales from millennia to years, from world wars and genocides to the spanking of children and the treatment of animals.

The fact that violence is so pervasive in history, but nonetheless can be brought down, tells us that human nature includes both inclinations toward violence and inclinations toward peace – what Lincoln called "the better angels of our nature" – and that historical changes have increasingly favoured our better angels. These changes include the development of government, commerce, literacy, and the mixing of ideas and peoples, all of which encourage people to inhibit their impulses, expand their empathy, extricate themselves from their parochial vantage points, and treat violence as a problem to be solved rather than as a contest to be won.

JN: I have to confess that my first reaction to the book was irritation – with myself. The case it makes seemed so compelling that I felt it should have been obvious to me beforehand. After all, it wasn't as though we didn't know about, say, the technology of the Inquisition, the rack, the thumbscrew etc. What annoyed me was that I had never made the connection between that abhorrent past and our relatively civilised present.

When I started to read the first chapter, for example, what came immediately to mind was a famous passage from Samuel Pepys's diary for Saturday, 13 October, 1660: "To my Lord's in the morning, where I met with Captain Cuttance, but my Lord not being up I went out to Charing Cross, to see Major-General Harrison hanged, drawn, and quartered; which was done there, he looking as cheerful as any man could do in that condition. He was presently cut down, and his head and heart shown to the people, at which there was great shouts of joy."

Pepys goes on to describe the rest of his day as if nothing untoward had happened. He goes to the Sun tavern with two friends, and buys them oysters. Then he goes home by boat, is so annoyed by his wife's untidiness that he kicks – and breaks – her basket, after which he spends the afternoon setting up bookshelves in his study.

I've always been struck by that diary entry (and was pleased to find it on page 145), because the sight of someone being hung, drawn and quartered would have traumatised any one of us. Imagine seeing something like that on your way to work! And yet Pepys, having been exposed to it, is able calmly to go about his daily business, and the crowd gives "great shouts of joy" upon being shown Harrison's heart. All of which makes your point – that cruelty we would now regard as unimaginable was once considered unremarkable.

SP: Though as a psychologist I like to think that nothing human is foreign to me, I admit to having been repeatedly flabbergasted by the insouciance, and sometimes relish, with which our ancestors carried out and witnessed unspeakable cruelties.

JN: One of the most intriguing tables in the book is the one on page 195, which takes the death toll from distant atrocities, estimates what proportion of the contemporary population that toll represents and then computes what the corresponding proportion of the mid-20th century population would be. So the Mongol conquests of the 13th century, for example, killed 40 million people, which corresponds to 278 million in 1950s terms. Now although 40 million is obviously a huge number, converting it into its modern equivalent makes one see it in a rather different light. Somewhere in the book you make a similar point about 9/11: the attacks killed 3,000 people, which of course is terrible at one level. But as a proportion of the US population, the death toll from the attack on the Twin Towers is, relatively speaking, infinitesimal. (That doesn't mean, obviously, that those deaths did not have a devastating impact on the friends and families of those who died.) So maybe one reason why we have such a warped historical perspective on the history of violence is down to what you call "the innumeracy of our journalistic and intellectual culture"?

SP: I think that a failure of statistical thinking is the major intellectual shortcoming of our universities, journalism and intellectual culture. Cognitive psychology tells us that the unaided human mind is vulnerable to many fallacies and illusions because of its reliance on its memory for vivid anecdotes rather than systematic statistics. Yet pundits continue to hallucinate trends in freak events, like the Norwegian sniper (who shot all those young people on an island) and make wildly innumerate comparisons, such as between Afghanistan and Vietnam, or between today's human trafficking and the African slave trade. It's a holdover of the literary sensibilities of our science-flunking intellectual elite, who would be aghast if someone didn't know who Milton was, but cheerfully flaunt their ignorance of basic science and mathematics. I lobbied – unsuccessfully – for a course requirement at Harvard in statistical and logical reasoning.

JN: I thought that your description of the Bible as "a wiki" is neat and your bleak recounting of the approved levels of violence in the Old Testament was sobering even to this atheist. It reminded me of anthropologist Edmund Leach's essay where he looked at the Bible through the eyes of a communications engineer and asked: what message are these authors trying to get through to the reader? The answer, Leach thought, was that they were trying to obscure the fact that mankind began through incest (Adam and Eve) and so the strategy was to compile a list of atrocities so heinous that, in the end, the original incest would come to look like a harmless act. I guess your portrayal of the Holy Book will get you into trouble in parts of the US.

SP: No doubt – though American evangelical Protestants are pretty un-intellectual, and are unlikely to read the book, and the conservative religious brains trust are mainly Catholic thinkers, who are less committed to the literal truth of the Bible. Nonetheless, they will not be happy with my claim that the Enlightenment (of which they are highly suspicious) was a very good thing.

JN: A big puzzle for me was an apparent contradiction between the argument of the book and the thrust of your earlier book, The Blank Slate, where you seemed to me to be arguing that, in understanding human behaviour, nature always trumps nurture. In The Blank Slate you're very critical of the theory that we are wholly shaped by our environment, and of the belief that humans are innately good but are corrupted by society. If I read you correctly, it all comes down to our genetic inheritance as shaped by evolution. But in Better Angels you seem to be arguing something quite different – that whatever our genetic composition, our environment and culture have, over aeons, altered our behaviour in very significant ways. Have I misunderstood you somewhere along the line?

SP: Yes, I think so. The Blank Slate did not argue that nature always trumps nurture, nor that it all comes down to our genetic inheritance. Rather, it argued that in order to understand the influence of culture, experience, and socialisation, we should get rid of metaphors such as society writing on blank slates or shaping lumps of putty, and identify the innate emotions, motives, and learning mechanisms of the human brain that allow people to create, share and acquire aspects of culture. That's the strategy I pursue in Better Angels – I try to explain historical changes in violent behaviour by showing how a changing historical environment (anarchy versus government, illiteracy versus literacy, parochial versus cosmopolitan milieus) engages the various parts of human nature in different ways. Sometimes our inner demons are activated, such as with greed, revenge, dominance, puritanism; at other times our better angels, such as self-control, empathy, fairness and reason have the upper hand.

JN: At the end, I was driven to ponder the attractiveness of the idea of decline in western culture. It's strange that the more "civilised" people become, the more convinced they are the world is going to the dogs. If your book provokes hostility from the chattering classes, it might be that it challenges this cultural narrative.

SP: Yes, a hostility to modernity is shared by ideologies that have nothing else in common – a nostalgia for moral clarity, small-town intimacy, family values, primitive communism, ecological sustainability, communitarian solidarity, or harmonies with the rhythms of nature. Presumably none of the romantics would want to go back to dentistry-by-pliers or biting a stick for pain management during surgery, but they point to today's wars and shootings to imply that we're still worse off today. Showing that war and homicide are in fact in decline undermines even that indictment of modernity.

Steven Pinker will give a talk at the Royal Geographical Society, London SW7, on 1 November. See intelligencesquared.com