Behind every coalition promise to "get tough on single mothers", behind every Daily Mail story about Britain's "handout culture", or Mitt Romney's notorious comments about "the 47%", there lies an assumption: that being poor is a failure of character. Awkwardly, for those who find this obnoxious, the research sometimes makes it seem true. People who are less well-off really do appear to give in more readily to temptation, making the very purchases they can't afford; to make unwise financial decisions; to use less effective parenting techniques; or to fail to take life-saving drugs, even when they're free. Is this a deep-seated weakness of will, made worse by a "culture of dependency"? The Harvard economist Sendhil Mullainathan and the Princeton psychologist Eldar Shafir reject that idea, and some of the most familiar leftwing responses, too. Poverty, they argue, is indeed a matter of willpower and bad decisions, but the Mail has it back-to-front. It's not that foolish choices make you poor; it's that poverty's effects on the mind lead to bad choices. Living with too little imposes huge psychic costs, reducing our mental bandwidth and distorting our decisionmaking in ways that dig us deeper into a bad situation.

Of course, it's hardly news that poverty creates a vicious cycle. Not having money is expensive, thanks to credit card late fees, high interest rates on payday loans, the extra cost of buying in instalments, and so on. But the alarming conclusion of this book is how completely scarcity colonises the mind. Merely asking poorer people to contemplate a hypothetical £1,000 car repair, one study by the authors shows, impairs their performance on intelligence tests as much as missing a night's sleep – about 13 or 14 IQ points. In another study, Indian sugar cane farmers performed worse pre-harvest, when money was tight, compared to post-harvest. "Scarcity captures the mind," explain Mullainathan and Shafir. It promotes tunnel vision, helping us focus on the crisis at hand but making us "less insightful, less forward-thinking, less controlled". Wise long-term decisions and willpower require cognitive resources. Poverty leaves far less of those resources at our disposal.

Their most arresting claim is that the same effects kick in – albeit not always with such grave implications – in any conditions of scarcity, not just lack of money. Chronically busy people, suffering from a scarcity of time, also demonstrate impaired abilities and make self-defeating choices, such as unproductive multi-tasking or neglecting family for work. Lonely people, suffering from a scarcity of social contact, become hyper-focused on their loneliness, prompting behaviours that render it worse. In one sense, Mullainathan and Shafir concede, scarcity is so ubiquitous as to be almost meaningless. But the feeling of scarcity – of not having as much of something as you believe you need – is something more specific and agonising. To use the authors' favourite metaphor, life under such conditions is like packing a tiny suitcase for a trip. It entails a ceaseless focus on difficult trade-offs: the umbrella or the extra sweater? The greatest freedom that money can buy is the freedom from thinking about money – or, to quote Henry David Thoreau, "a man is rich in proportion to the number of things he can afford to let alone".

There's a risk here of lapsing into the obvious: rich and relaxed is better than poor and time-starved. Mallainathan and Shafir do sometimes succumb; financial abundance, we are gravely informed, "allows us to buy more things". Yet the strongest chapters demonstrate that the psychological effects of scarcity aren't obvious at all. In certain limited ways, for example, poverty actually confers cognitive benefits. Some of the classic findings about how irrational we are when it comes to money – such as our willingness to travel across town to save £5 on a cheap toaster, but not on a flatscreen TV – apply much less to the poor. Dieters, experiencing a scarcity of food, are significantly better than others at identifying words briefly flashed on a screen, provided that they're about food. Lonely people read facial expressions more accurately. And time-scarcity brings motivational benefits, as any journalist on a deadline could tell you.

But these positive effects of tunnel vision are outweighed by what the authors call "the bandwidth tax", the ways scarcity limits or distorts our skills. This tax, they argue persuasively, explains a number of otherwise confounding kinds of self-defeating behaviour among those suffering scarcity – from the failure of poorer farmers in Africa to weed their fields, even though they have the time to do so and would make more money that way, to the failure of low-income Americans to take diabetes drugs and other medications, or to eat more healthily even when it's financially viable. "The failures of the poor are part and parcel of the misfortune of being poor in the first place," they write. It's not that poor people have less bandwidth. It's that "all people, if they were poor, would have less effective bandwidth".

The bandwidth argument threatens to undermine much received political wisdom on poverty. Get-tough policies, like cutting off access to benefits after a fixed number of years, won't motivate people to find jobs: a deadline of several years is too distant to feature in the calculations of people only concerned with paying the next bill. On the other hand, well-intended interventions like providing financial education or job-readiness training could backfire, too. Another class to attend, another item to tick off the to-do list – all use up more bandwidth, potentially impairing people's capacities more than improving them.

How can we stop falling into these traps? Mullainathan and Shafir offer a few "nudge"-style suggestions. Where possible, systems should be designed so that inattentiveness leads to better outcomes, for example by making savings schemes opt-out, not opt-in. Beneficial behaviours could be "brought inside the tunnel": the authors describe their own experiments with an "impulse savings" scheme, involving cards sold at supermarket tills, resembling gift vouchers, but which credit the purchaser's savings account instead. And behaviours that require constant, energy-depleting vigilance (like trying to resist non-essential spending) should be replaced by one-off actions (like automatically transferring a percentage of your wages to a savings account). But they wisely don't pretend to offer a comprehensive solution. The tendrils of scarcity reach too deep into the mind. Poor people need more money, not self-help tricks.

The overall result is a rather odd but ultimately humane and very welcome book. Presenting itself as yet another "big idea" tome that will reveal the unexpected force that explains the world, Scarcity ends up reaffirming one of the oldest truths: that what really explains the world is its division into haves and have-nots. The clear message to those with resources – money, time, or anything else – is to resist the urge to judge those without them. If you faced the same scarcity, Mullainathan and Shafir demonstrate, you'd make the same mistakes. Indeed, in some area of your life – if not your spending, then your work/life balance or your diet – you're almost certainly already doing so.

• Oliver Burkeman's The Antidote is published by Canongate.