Have we reached the end of progress? Today, this lodestar of the Enlightenment and the Industrial Revolution is shining less brightly than at any time in the past 200 years. And our principal measure of progress – gross domestic product (GDP) – seems particularly tarnished. Growing numbers of people are asking whether economic growth, measured by whether GDP is going up, should be the main priority of governments. Aren’t the environmental costs of growth too high? Is higher GDP worthwhile if it all goes to the rich? Does higher GDP even make us happier, and isn’t that what our governments should really focus on?

All these questions end up tangled in statistical definitions, obscuring the deeper issues they raise. Some of the questions are philosophical. I, for one, don’t want the government dabbling in my well-being. And I do think that economic growth is a sign of progress, but I want it to be more equally shared than has been the case in the past generation. Some of the issues are more technical: what are the binding environmental constraints on growth and how do we measure them? None of these questions can be answered by changing the definition of GDP.

The measurement we now know as GDP (related to the original measure, GNP, or gross national product) was developed by economists in the UK Treasury, working under the guidance of the eminent John Maynard Keynes, and completed in 1941. One of these economists, Richard Stone, led the post-war initiative to spread GDP as a standard within the United Nations. In just a matter of years, national economic accounts were progressively standardised, bureaucratised, and adopted around the world for the remainder of the 20th century. GDP is now the universal benchmark of economic standing.

The case made against GDP is that it does not measure what we truly value. While counting economic activity such as output of cars or meals out, GDP leaves out non-monetary costs such as pollution; it ignores inequality and work-life balance, and counts ‘bads’ (more lawyers, more weapons purchased) as ‘goods’. A typical critic of GDP is Michael Green, a US economist and executive director of the Social Progress Imperative in Washington DC. This organisation created the Social Progress Index (SPI) to measure a nation’s well-being by looking at non-economic indicators such as whether people have access to basic medical care and higher education. As Green told US News last month: ‘There’s lots and lots [GDP] doesn’t capture. [It] doesn’t tell us about the quality of our lives in terms of the real things that matter to real people.’

However, it was never meant to do this, so this failure should not come as a surprise. But there is a profound confusion about what GDP does measure, and about what should be measured, and why.

GDP is simple in principle: it is the sum in a given time period of everything produced in the economy with a monetary value, which should add up to the same figure as the incomes earned by every person and company, and the same as the total spent by everyone. In practice, these separate sides of the accounts are rarely equal because of the difficulty of assembling all the vast sets of statistics. National accounting is an esoteric art that, four times a year, delivers us a figure – up 0.2 per cent! Up 0.6 per cent! – trumpeted (or not) by governments keen to show their constituents that they are delivering.

Criticisms of GDP date back to the 1970s, when high inflation and zero growth (‘stagflation’) led to widespread economic disillusionment. Capitalism did not seem to compare well with Communism. Economic growth had not taken off as hoped in the former colonies. And, perhaps most profoundly, the emerging environmental movement began campaigning about the costs economic growth inflicted on nature.

Environmental sustainability remains central to criticisms of GDP today. The measure ignores the adverse impact of pollution, greenhouse gas emissions, reduced biodiversity, and the depletion of natural resources. And it fails to adjust for the fact that spending on policemen or lawyers is a necessary evil not a positive benefit, or that we spend more time commuting these days, or that the infrastructure of roads and railways is crumbling. GDP-bashers also delight in pointing out that, over time, increasing GDP has not delivered increasing happiness; happiness, as reported in many surveys, has, they claim, risen far, far less.

This last point misconstrues the nature of two types of statistic. GDP is an artificial construct, not a natural object, and it can rise without limit. ‘Happiness’ is a state of mind, typically measured by psychologists on a scale of 0 or 1 to 3, or 1 to 10. But, just as the growth of GDP has contributed to longer lives and taller humans, so it has also contributed to making us happier – there’s plenty of evidence that richer people are consistently happier than poorer people. In that sense there’s a positive correlation.

Still, the emphasis on happiness, or ‘positive psychology’, like that on environmental sustainability, raises some important questions about the purpose of economic policy. To explore these, we need to go back to the origins of GDP, and look at why it was devised in the first place.

GDP is only the latest of a number of approaches to measuring the economy, dating back to the late 17th century when ‘the economy’, as we now think of it, started to become a meaningful idea. Interest in measuring it has always been driven by the interests of the state. In the days of William Petty – the pioneer of national accounting, who produced the first version of GDP for England and Wales in 1665 – the motive was to assess the country’s capacity to pay taxes that would finance expensive wars against Holland and France.

Tax revenues and trade remained the main interest behind measuring growth throughout the 18th and 19th centuries, but the concept of ‘the economy’ changed markedly over that time. Adam Smith, for example, insisted in Wealth of Nations (1776) that growing food and manufacturing goods created true wealth, which, in turn, paid for services with no inherent value. His concept of ‘material’ production remained the foundation of economic statistics in the Soviet bloc until 1989.

In the West, government interest in an aggregate measure of a nation’s economy was reignited by the Great Depression, when the chief question was: how much idle capacity is there? How many factories are shuttered, and how much more could they produce if put back to work? In the UK, Colin Clark took the lead, as statistician to the new National Economic Advisory Council in 1930, and Simon Kuznets was one of the economists leading this work in the US.

Kuznets had firm ideas about how to define the economy. He thought the point of measuring national income was to understand how much of the good things in life private individuals could consume, both now and through their future savings. How well did the economy serve their well-being or ‘welfare’?

Kuznets’s first report, submitted to Congress in January 1934, showed that the size of America’s economy had halved between 1929 and 1932. Even in those depressed times, so great was the interest in reviving economic growth that the report was a best-seller. President Roosevelt cited Kuznets’s figures when announcing the new Recovery Program in 1933, and updated them before sending a supplemental budget to Congress in 1938. Having national income estimates for the whole economy made a huge difference to the scope for policy. President Hoover before him had made do with the incomplete picture painted by industrial statistics, such as share price indexes and freight car loadings. This information was less compelling as a call to action than authoritative figures showing the halving of national economic output in the space of just a few years.

GDP is meant to measure market activities and, by definition, housework is not in the market; but there’s no market price for government activities either, and they are included

Kuznets, however, saw his task specifically as working out how to measure the total amount produced in the economy, by adding up the output of all the different industries and the amount consumed by private individuals that brought them some benefit. To this end, he wanted to subtract from the national income all expenses relating to armament, to financial and speculative activities, and the enormous outlay relating to transport and housing. These expenses, he claimed, ‘do not really represent net services to the individuals comprising the nation but are, from their viewpoint, an evil necessary in order to be able to make a living.’

Kuznets’s idea of national income referred not to economic activity or output (in other words, GDP) but what can be described as ‘economic welfare’, a measurement of well-being. As he said in 1934: ‘The welfare of a nation can scarcely be inferred from a measurement of national income.’ This is the same criticism of GDP made by its many critics now, when they point out that we’d do better to have a measure that deducts things such as environmental costs, or the costs of preventing crime or fighting wars.

However, in the debate about whether we should be measuring welfare or economic activity, Kuznets lost. The Second World War made the latter more important. The US and UK governments wanted to know how much war material their economies could produce, and what sacrifice that would mean for the civilian economy. Welfare or well-being (as it is often referred to now in the economics literature) looked like a peacetime luxury.

The universal standard definition of GDP embodies certain assumptions about what stays in and what stays out of the measure. So, it includes government consumption, in line with Keynes’s macroeconomic theory which views government spending as a form of collective consumption. But it omits unpaid work in the home, or caring for family members. GDP is, after all, meant to measure market activities and, by definition, housework is not in the market; but then there is no market price for government activities either, and they are included.

Another assumption concerns how to measure finance. Post-war statisticians included those aspects of finance that clearly provided a service with a market price attached, such as advisory and management fees, but left out trading activity. (Adam Smith would have left out the financial sector entirely.) Owing to boundary creep, more and more financial activities were included as the sector grew. The upshot is that the biggest contribution to the UK’s GDP from the financial sector occurred in the final quarter of 2008, which is clearly absurd.

Measuring GDP has grown increasingly complicated as the economy itself has become more complex. It is relatively easy to count how many more cars or computers are made and sold each year. In the mass assembly era, bigger numbers were a good measure of economic growth. But what about in the ‘new economy’, where the quality and power of products is increasing so dramatically? Should GDP count how many laptops are sold; or should it acknowledge their increased speed and storage, the built-in Wi-Fi and cameras, their lighter weight? GDP statistics now have very many complicated adjustments made to take account of quality enhancement. These so‑called ‘hedonic’ adjustments are made to a range of consumer electronic goods, but not to many other items counted in GDP – it does not capture the greater effectiveness of painkillers, energy-efficient light bulbs, ‘intelligent’ fabrics, or the much greater consumer choice across the board.

Yet, as a 1998 report from the Dallas Federal Reserve Bank put it: ‘GDP is a statistic designed for mass production. It’s a simple counting – the number of units made. It falls short in measuring intangible benefits… Nobody ever said quantity was the spice of life.’

For all these reasons – the exclusion of some but not all unpaid activities, changing definitions, the omission of innovation and variety – there’s no doubt that GDP is a flawed measure of economic activity. However, critics of GDP are not usually thinking about these drawbacks. They’re really calling for a measure of the change in welfare or well-being, rather than a measure of economic growth. Being clear about the difference is important for working out what to measure in future.

Kuznets lost the argument about measuring social welfare over economic activity back in the 1930s. Now it’s time to put these two concepts in the balance again, not least because so many critics of GDP fault it for not measuring well-being. It was never meant to. Yet while we have always known this, economists have routinely used GDP growth as shorthand for well-being. And while this has a sound rationale, there are good reasons for thinking that the gap between social welfare and economic activity, as measured by GDP, is widening.

a natural disaster spurs faster GDP growth because of the reconstruction it necessitates, but the loss of life and assets will not be included

While GDP does not begin to capture the gains to well-being stemming from innovation, new medicines, smartphones or graphene, zips or dishwashers, or fabrics that are easy to clean, there’s plenty of evidence to suggest that people greatly value increased choice, even in seemingly trivial products such as the flavour of breakfast cereals or – less trivially – the range of book titles published. And innovation is why economic growth is necessary for progress: economic growth is innovation. But, as the critics always point out, GDP growth also overstates well-being, by not incorporating environmental externalities, or the increased inequality of incomes, and by counting ‘bads’ as ‘goods’; a natural disaster, for example, spurs faster GDP growth because of the reconstruction it necessitates, but the loss of life and assets will not be included.

Does it make sense, then, to ‘adjust’ GDP to acknowledge this range of both positive and negative omissions? That path seems wholly misguided to me. There’d be endless arguments about what to include and exclude, and what weights to give different elements of a new GDP definition. How do you trade off the plus of smartphones against the minus of fracking? Or the plus of cleaner air against the minus of a narrower range of wildlife?

More fundamentally, we need to keep distinct the two concepts of economic activity and well-being, and more aggressively direct policy towards the latter. One option, currently practised by the UK’s Office for National Statistics (ONS), is to measure well-being directly through an annual survey. However, since the ONS began collecting data in 2012, the results do not seem to have changed much, and it’s not clear what policies might increase the survey findings anyway. Research so far suggests that sex, friendship, and being part of a faith community are positively linked with reported happiness; while long commutes, unemployment and mental ill health link negatively. That said, you don’t need an annual survey to tell the government to keep unemployment low, or fund mental health services adequately. And a policy for more sex is hard to imagine.

Many of the things we value are notoriously difficult measure in monetary terms, and add up. So why try? Organisations such as the OECD with its Better Life Index, and the Australian Bureau of Statistics, have been working on ‘dashboards’ of the indicators of well-being, including economic indicators such as income and jobs, but also the quality of the natural and urban environment, work-life balance, health, engagement in the local community, and access to education. The dashboards are far from perfect but they could eventually become just as important as GDP growth for policymakers to monitor over time. They also reveal the hard choices, or trade-offs in policy-wonk speak, because we can’t have everything.

The most important trade-off to measure, and report to citizens, is that between the present and the future; in other words, sustainability. No adjustments to GDP will ever capture this crucial factor because it measures activity during a certain period of time (it is a ‘flow’). By contrast, assessing sustainability depends on the depletion or accumulation of assets. And the statistics needed to do that – for environmental assets as well as physical ones such as bridges or the housing stock – although collected in today’s national accounts, are never reported in the headline-grabbing way that will reorient government policy.

This kind of measurement would tell us if we were enjoying the fruits of economic growth (higher GDP) only by eating up tomorrow’s capital. If all the innovation and creativity encapsulated in the measurement of growth proves unsustainable, then we might well be looking at the end of progress. But we do not yet know the answer, and we will not find out by changing or scrapping GDP.

GDP: A Brief But Affectionate History (2015) by Diane Coyle is published via Princeton University Press.