David Brooks is aware there’s some irony in the subject matter of his latest book, which is a hymn to humility, and the importance of acknowledging how little we can ever truly know. As a twice-weekly opinion columnist for the New York Times, and a fixture on US television and radio, he is, in his own words, “paid to be a narcissistic blowhard, to volley my opinions, to appear more confident about them than I really am”; he is also one of the few conservatives whose views Barack Obama often solicits. Best known to British audiences for his books Bobos in Paradise and The Social Animal, Brooks says he considered calling the new one Humility, with the title in tiny letters and “David Brooks” in huge block capitals. In the event, he called it The Road to Character, which seems unlikely to mollify the army of bloggers who appear to find him insufferably pompous – “the biggest windbag in the western hemisphere”, to quote Rolling Stone’s Matt Taibbi. (New York magazine once described Brooks’s role in American life as “public intellectual/punching-bag”.) This is a pity, because it’s a powerful, haunting book that works its way beneath your skin. And the motive for writing it, far from being a pompous desire to sermonise, was at least partly due to a personal crisis: Brooks’s realisation that his own life of well-paid worldly success, plus regular meetings with the president, was missing something essential inside.

“I started out as a writer, fresh out of college, thinking that if I could make my living at it – write for an airline magazine – I’d be happy,” says Brooks over coffee in downtown Washington, DC; at 53, he is ageing into the amiably fogeyish appearance he has cultivated since his youth. “I’ve far exceeded my expectations. But then you learn the elemental truth that every college student should know: career success doesn’t make you happy.” In midlife, it struck him that he’d spent too much time cultivating what he calls “the résumé virtues” – racking up impressive accomplishments – and too little on “the eulogy virtues”, the character strengths for which we’d like to be remembered. Brooks builds a convincing case that this isn’t just his personal problem but a societal one: that our market-driven meritocracy, even when functioning at its fairest, rewards outer success while discouraging the development of the soul. Though this is inevitably a conservative argument – we have lost a “moral vocabulary” we once possessed, he says – many of the exemplary figures around whom Brooks builds the book were leftists: labour activists, civil rights leaders, anti-poverty campaigners. (St Augustine and George Eliot feature prominently, too.) What unites them, in his telling, is the inner confrontation they had to endure, setting aside whatever plans they had for life when it became clear that life had other plans for them.

This may seem like serious stuff to readers who recall Bobos in Paradise, Brooks’s acutely well-observed debut about the new class of “bourgeois bohemians”, the emerging elite who fused the social values of the hippies with the consumerism of the yuppies. Returning to America following a stint in Brussels for the Wall Street Journal, Brooks found that it was “now impossible to tell an espresso-sipping artist from a cappuccino-gulping banker.” (He counted himself a Bobo, as must at least some Guardian readers.) He zeroed in on a new form of conspicuous consumption: a Bobo would never spend thousands on a fancy TV – that would be crass – but would willingly blow cash on “necessities”, such as restaurant-quality kitchen appliances, or a bathroom lined with slate of precisely the right artisanal roughness. For Bobos, he explained, it was “perfectly acceptable to spend lots of money on anything that is of ‘professional quality’, even if it has nothing to do with your profession”: if you’ve ever purchased an “expedition-weight, three-layer Gore-Tex Alpenglow reinforced Marmot Thunderlight jacket” for a country hike, as opposed to an Everest ascent, he meant you.

Each of his subsequent books probed humanity slightly more deeply: On Paradise Drive sought to defend US consumerism as a form of self-expression; then The Social Animal drew on psychology and neuroscience to explore human nature and the modern epidemic of loneliness. Even so, The Road to Character feels like an abrupt plunge that goes far deeper. Though not explicitly religious, Brooks’s language evokes theology: for example, he doesn’t shirk from using the word “sin”, not in a scolding sense, but to refer to the universal tendency to “get our loves out of order”, prioritising what doesn’t matter most. A friend in publishing, hearing him speak about the book while he was writing it, called Brooks and said: “Do not use that word ‘sin’ – it’s so off-putting!” But Brooks concluded that it was necessary. “Sin isn’t the Holocaust; sin is spending your life thinking more about how you’re coming across [in a conversation] than on what the other person’s saying. These kinds of small sins that we do every day.” His point is not simply that we’re too focused on money, fame or possessions. Even someone committed to doing good – working for good causes, raising children well, helping the community – can too easily end up skipping the internal work of confronting our weaknesses, our inherent “brokenness”, required to achieve the richest inner life.

The journalist Michael Kinsley once wrote that “like the elderly Jewish lady who thinks someone must be Jewish because ‘he’s so nice’, liberals suspect that a writer as amiable as Brooks must be a liberal at heart”. Conservatives often accuse him of not being conservative at all. Yet a deep conservatism is implicit in his concentration on the inner life: he is highly sceptical as to whether government can do very much to help us live better lives. (He quotes Samuel Johnson: “How small, of all that human hearts endure, that part which laws or kings can cause or cure.”) The result can infuriate liberals: one recent column attracted scorn for arguing that the poor needed to recover long-lost social norms – “basic codes and rules woven into daily life” – as much as they needed, say, more money. He has written of the “materialist fallacy” that economics determines the fates of societies, arguing that culture and character matter as much. And his repeated emphasis on the US’s “dangerous level of family breakdown” meant eyebrows were raised when it emerged that he and his wife of 27 years, with whom he has three children, were divorcing. “Scant self-awareness” was the charge made by Matthew Yglesias, writing in Slate.

Yet, if anything, Brooks has a surfeit of self-awareness: more than any similarly high-profile columnist, he has used his public writing to wrangle with his own doubts, ambivalence and self-reproach. In a particularly scathing 2013 column, entitled The Thought Leader, he excoriated the typical older member of the pundit class – in which he plainly included himself – as a “yacht-to-yacht concept pedlar”, “bruised by snarky comments from new versions of his formerly jerkish self … his cries for civility, and good manners [are] really just pleas for mercy to spare his tender spots.” His most caustic columns are usually about his own flaws, he says, adding disarmingly: “I’ve made a career out of self-hatred.”

He did start out emphatically on the left. Raised in New York by Jewish parents both in academia, he was “a socialist” at the University of Chicago – sufficiently so that he was among several students hired for a TV debate with Milton Friedman, the legendary rightwing economist. (As Brooks admits, and as the clips on Youtube demonstrate, Friedman skewered him with embarrassing ease, striking the earnest young student dumb.) After graduation, he worked as a police reporter in Chicago, and attributes his rightward shift to “seeing the unintended consequences of the city’s social policies” in its notorious public housing projects. That experience – along with a healthy concern for career advancement, presumably – propelled him to an internship at the National Review, whose founder, the conservative eminence William F Buckley, took him on as a special project. “For 18 months, he took me yachting, took me to Bach concerts, took me to dinners where I was the youngest person by 40 years,” he recalls. Still, it was an awkward fit for Brooks, who was never a social conservative on issues such as gay rights and abortion, as were Buckley’s Catholic circle. He left for the Wall Street Journal, and then the newly founded neocon magazine, the Weekly Standard. A few years later, he was approached by the New York Times’s comment editor, Gail Collins. “I was looking for the kind of conservative writer that wouldn’t make our readers shriek and throw the paper out the window,” she told an interviewer. “He was perfect.”

Thus Brooks mounted the treadmill of finding two opinions per week to air in the nation’s most influential paper, despite describing himself as someone who finds it hard to generate opinions on demand. He says: “I once had lunch with a prominent American columnist – I won’t say who – and I asked: ‘What’s your next column about?’ He pulled out an index card from his wallet, and the next 13 columns were on there. I wanted to take my knife and ram it into his neck.”

Despite his being hired as a conservative who wouldn’t make liberals shriek, some do, especially when he writes on foreign policy. Brooks’s willingness to try out ideas in public, then change his mind later, may be admirable when he’s waxing philosophical – but it’s another thing when the topic is, say, the Iraq war, which he strongly backed before reversing course. His early days at the paper, he has said, were “the worst six months of my life. I had never been hated on a mass scale before.” His support for Obama helped establish him as the incoming president’s favourite conservative: the night before each column appeared, New York magazine reported, he’d get a call from a senior aide asking if the next day would be a good one, meaning: would Brooks praise or criticise the White House? (In fact, it was the critical columns that triggered the personal invitations: Obama wanted to persuade him he was wrong.) “I was super-impressed by him,” Brooks says. “And I would still say that he is a man of great character. He’s more leftwing than I thought he was, so I disagree with him more, but I don’t disrespect him more.”

He is less enthusiastic about politics in general, though, in keeping with the new book’s inward turn. “I just find talking to politicians less interesting than I used to,” he says. “I used to find it fascinating, what all the little subterfuges going on in the construction of the immigration bill were. But I just can’t get my interest up any more. There’s a lot more action sociologically, psychologically, morally than politically, these days.” He lights up when asked to discuss the course he teaches to Yale University undergraduates, one day a week, entitled Humility. (Cue more blogosphere jokes.) “To get into a place like Yale, you have to work so hard, and these students know they’ve not spent time on this other part of their lives – so like any normal person, they feel a dryness, or a shallowness.” After completing the set readings – Augustine, Homer, Montaigne, the theologian Reinhold Niebuhr – they have “a new set of categories, a new set of things to worry about”. Recently, one student told him that, since taking the course, he was much sadder than he used to be. “That’s a high compliment!” says Brooks. “He was a phenomenally bright and successful student. But, you know – you should be a little sadder, sometimes.”

• The Road to Character is published by Allen Lane on 14 April