When David Brooks was a boy, he had two turtles named Gladstone and Disraeli. How come? "There's a New York Jewish culture that has a saying 'Think Yiddish, act British'," says Brooks. "My background was filled with Anglophile Jews. Jews of a certain generation, really my grandfathers' generation, gave each other names they thought would help them fit in – Irving, Sydney, Milton and Norman – and now in the US those are not English names any more, they're Jewish names. And I was brought up in that culture. Hence the turtles."

Hence much more than that. Brooks, though a 49-year-old Canadian-born, suburban New York-raised, Chicago university-educated and now so much of a stellar New York Times columnist that the White House sometimes rings him to ask what he's planning to write about, is deeply Anglophilic.

"I am very British in that I'm reticent. There's a survey of how many times people in different countries touch each other during an hour over coffee. In Rio it was 180, in Paris 120. London, zero." How about New York? "Maybe 40? I feel very at home here." We're sitting in the Cinnamon Club, an Indian restaurant in Westminster frequented by policy wonks, and he looks more diffident than the only Englishman at our table. I resist the counter-cultural urge to play footsie.

But what's important about Brooks is not so much that he acts British, but that he thinks British. His new book, The Social Animal: A Story of How Success Happens, is steeped in the anti-rationalist philosophical reflections of the British Enlightenment. And this is no ordinary book: even before publication this week it has become, according to Times columnist Rachel Sylvester, "the must-read text for politicians searching for a new prism through which to examine the apparently intractable challenges of social immobility, school dropout rates, welfare dependency and crime". Education secretary Michael Gove believes it contains vital clues for turning around failing schools; universities minister David Willetts reckons it may help define modern Conservatism; policy minister Oliver Letwin thinks it articulates the cherished Tory notion of the Big Society. The book is so hot that both David Cameron and Ed Miliband are meeting Brooks this week, and Steve Hilton, the PM's top strategist, has invited him to hold a seminar at No 10 on Friday.

Brooks hails British rather than French Enlightenment thinkers as the guys who really understood what makes the social animal tick. While Voltaire, Condorcet and Descartes used reason to confront superstition and feudalism, thinkers across the Channel – Brooks cites Burke, Hume and Adam Smith – thought it unwise to trust reason. Rather, and here Brooks quotes Hume with approval: "Reason is and ought to be the slave of the passions."

Why is The Social Animal so important if it just dusts off old thoughts of Brits from 200-plus years ago? First, Brooks argues misplaced faith in human rationality has underpinned policy-making for too long. Second, research in neuroscience, behavioural economics and psychology stressing the importance of our non-rational minds can, if applied, create a better world.

Brooks says that, overwhelmingly, human decision-making is not rational but unconscious. Much of the book's pleasure consists in reading digests of experiments (such as international differences in the incidence of touching during coffee) that show how non-rational we are and yet how successful the social animal when breaking free of mere rational decision-making. The style and substance will be familiar to readers of pop psychology bestsellers such as Malcolm Gladwell's Blink or Jonah Lehrer's Proust Was a Neuroscientist: for Brooks the unconscious isn't a seething Freudian netherworld of sexual urges, but where we make the key decisions of our lives – whom to date and marry, how to vote.

Most success stories stress academic ability, IQ, hard work, he argues. Brooks rather stresses non-cognitive skills, which, he writes, is "the catch-all category for hidden qualities that can't be easily measured, but which in real life lead to happiness and fulfilment." "By that I mean emotions, intuitions, genetic inheritance. Soft stuff, which is pretty rich given that my wife thinks I'm insufficiently touchy feely."

And what are these mysterious non-cognitive skills? Good character (energy, honesty, dependability, recognising your weaknesses and controlling your worst impulses). He also mentions "street smarts", by which he means reading situations and people, often unconsciously, and developing human relationships. He thinks these skills can be honed.

He gives examples of policy-making without non-cognitive street smarts. "When we invaded Iraq we were blind to the social problems that would be involved. We didn't realise they didn't trust us." Hold on – didn't he write a New York Times column urging invasion? "I did. I was so blind about it. In that column I wondered what Michael Oakeshott [the British conservative political philosopher] would have said. He would have said: this society is very complicated and you should be circumspect in thinking about what you can achieve, and that invading to install democracy without trust is doomed. And then I wrote: 'Having said that, I think we should invade.'"

Another example is the banking crisis, which, he reckons, happened because we trusted bankers. "Many thought we should let these rational wealth-seekers get on with it. We shouldn't."

The Social Animal's thesis is expressed through the form of a novel. He creates a couple, Harold and Erika, he from a rich background, she from a broken family in a disorganised neighbourhood, and traces them through their formative years, marriage, careers, retirement and death. The book has become a US bestseller and is worth reading – even if with mounting exasperation – since it seems to promise answers to some of western society's deepest problems: how to generate social mobility and reform a non-society devoid of mutual trust and bristling with security cameras.

No wonder leading Tories welcome Brooks. He is to the Big Society agenda what Richard Layard was to Labour's happiness philosophy and Richard Sennett was to Blair's respect agenda. "The Big Society appeals to me because I don't think appealing to people as individuals gets you far. Many social problems are caused by insufficient social capital. Kids are brought up in broken homes and crime-ridden neighbourhoods; they don't go to university because they're not attached to their schools . . . to solve these problems you need to build dense social networks. You have to get beyond treating people as rational machines who respond to the economic incentives."

Brooks thinks his book, written with the US in mind, speaks to British problems. He quotes the jeremiads of self-styled Red Tory Phillip Blond about Britain having become a bipolar nation in which a bureaucratic, centralised state presides over a fragmented, disempowered and isolated citizenry. "I get to where Blond is by arguing that there have been two individualist revolutions. Conservatives embraced the individualism of the market and reacted furiously if the state impinged on individual economic choice." Brooks writes that one consequence of this is chains such as Walmart closing local shops, destroying networks of community those shops created.

"There's also a liberal revolution in the moral sphere that says the state shouldn't impinge on choices about marriage, family structure, the role of women. That liberal revolution also took religion out of the public square. Together these revolutions undermine communal trust and law and order." It also, he writes in the Social Animal, led to welfare policies that "enabled lonely young girls to give birth out of wedlock, thus decimating the habits and rituals that led to intact families".

Perhaps the fact that you're a self-described socialist will appeal to Ed Miliband, I suggest to Brooks. "Yes, but my socialism doesn't value state over society. It favours a more communitarian style of politics. The point is to ensure that people from different classes feel united in a common enterprise. When I meet Ed Miliband, I might ask if my kind of socialism appeals, or if he's stuck with the old one."

My hunch is that Brooks's socialism would make Miliband queasy. In the book, he eulogises charter schools – schools that get public money but are granted autonomy from state control in exchange for producing certain results, notably targeting kids from tough backgrounds. Erika, his character from a tough background, manages to get to just such a school established by a billionaire hedge fund trader.

But aren't charter schools anti-egalitarian, don't they stop people from different classes feeling united in a common enterprise? "These schools are unequal, but in an unequal society you need that. Poor kids need different things from schools than rich kids because they often don't have the structure in their homes or neighbourhoods to give them a chance of success and most schools don't help with that."

Isn't there a risk that decentralisation undermines your socialism? "Yeah. What I want to say to David Cameron is that if you decentralise power you risk getting rid of a basic level of fairness and equality. And you risk creating separate communities that don't talk to each other." Brooks cites Guardian columnist Polly Toynbee who on Tuesday rounded on Eric Pickles's localism bill. "It was a good article because it argued that when budget deficits are cut the poor are at greater risk. Not that I'm saying cutting the deficit is wrong; it's right, but it needs not to fall on the poorest hardest."

Brooks tells me he is a fan of Anthony Trollope, something not admitted by a public figure since John Major. He recently gave a talk to New York's Trollope Society about the novel The American Senator. "In it the senator scorns British political institutions, arguing they're absurd and irrational. The Lords? Ridiculous. But what Trollope felt when he ridicules that senator, and what I share, is a belief in institutions to achieve communal goals and how wrong it is to try to impose rationalistic models on existence."

How un-American. Brooks reminds me of a reverse Jonathan Freedland. While Freedland's book Bring Home the Revolution argued the egalitarian ideas of American revolution should be imported to reform Britain's insufficiently rational polity, Brooks seems to be arguing that it doesn't matter that Britain's political institutions aren't rational, just that they need to be infused with more communal spirit and funky-sounding streets smarts. Whether that's a message Britain wants to hear is another matter.

• The Social Animal is published by Short Books, £14.99. To order a copy for £11.99 with free UK p&p, go to theguardian.com/bookshop or call 0330 333 6846.

Ten years of brain food

Patrick Kingsley on the influential books of the past decade

The Tipping Point, Malcolm Gladwell, 2000

Why did crime drop so dramatically in New York during the mid-1990s? And how does a book by a relatively unknown journalist end up as an international bestseller? In an age before Facebook and Twitter, New Yorker writer Gladwell explored how social oddities move so suddenly from obscurity to popularity.

No Logo, Naomi Klein, 2000

Published in the aftermath of the 1999 Seattle riots, No Logo attacked the unethical practices of large brands and corporations, and tapped into the consciousness of a growing anti-globalisation movement.

Freakonomics, Steven D Levitt and Stephen J Dubner, 2005

The work of an economist (Levitt) and a writer (Dubner), Freakonomics used economic theory and data to explain social phenomena. Gladwellian in approach, their book nevertheless disagreed with some of The Tipping Point's conclusions.

The Long Tail, Chris Anderson, 2006

Globalisation may have created a fairly homogenous consumer market, but alternative culture still has a future – or so argued Anderson, the editor-in-chief of Wired. There is, he wrote, a "long tail" of niche products that collectively attract a great deal of consumer interest.

The Black Swan, Nassim Nicholas Taleb, 2007

While contemporary ideologues tried to rationalise the unexplained with statistics, philosopher Taleb said such accounts could work only in hindsight, and would not help predict future surprises (what he terms "Black Swan events"). He advocates building a society that can limit the damage of Black Swan events once they inevitably occur.

The Terror Dream, Susan Faludi, 2007

In The Terror Dream, journalist and feminist Faludi analysed America's psychological reaction to the 9/11 attacks. She argued that in fracturing the myth of American invincibility, the attacks also indirectly prompted a resurgence in patriarchal ideals, and a return to old-fashioned perceptions of gender.

Here Comes Everybody, Clay Shirky, 2008

Here Comes Everybody showed how the web had democratised group interaction. Shirky, a theatre director turned internet evangelist, claimed communal websites such as Wikipedia made traditional institutions redundant, and predicted that bloggers would soon usurp mainstream news outlets as distributors of information.

The Spirit Level, Richard Wilkinson and Kate Pickett, 2009

Written in the midst of the financial crisis, The Spirit Level attempted to show how countries with wide income disparities tended to face more social problems – more crime, more violence, more drug abuse, worse education, and less social mobility.

The Idea of Justice, Amartya Sen, 2009

A revision of John Rawls's A Theory of Justice, Sen's book suggested that social justice is not a binary concept, but exists instead on a sliding scale. It inspired significant internal debate within New Labour – James Purnell and Liam Byrne were fans; Jon Cruddas was less convinced.

Delusions of Gender, Cordelia Fine, 2010

For years, books such as Men are from Mars, Women are from Venus peddled the belief that there are major neurological differences between men and women. In Delusions of Gender, Fine, an Australian psychologist and academic, suggested differences in behaviour between men and women have social rather than genetic causes.