A little after 3 pm, 16-year-old Jay Thompson walks from the first-floor school gym, where his locker is, up four flights of stairs to the school library, sits down at a computer, and sets about deciding what to do with his life.

On the screen in front of him is a program called ‘Career Voyage’. Every year, around 250,000 Australian Year 10 students will undertake some form of careers guidance. “These types of online programs are just the beginning,” explains Jay’s careers counsellor, Ken Tinnet. “The basic idea is to get students thinking about what activities they enjoy … and then help them join the dots to jobs.”

Jay’s school is on the outskirts of Wollongong, an industrial heartland in the Illawarra region of New South Wales. Three out of four students leaving Jay’s school will pursue a vocational career. Some plan to follow in their parents’ footsteps to become carpenters or welders or nurses. But many of the 16-year-olds who knock on Ken’s office door for a 20-minute pep talk about their future are anxious. They don’t know what they want to do. Plenty of students have a family member who has lost their job and struggled to find a new one.

To add to that, it seems every day brings with it a new story about how robots are set to take our jobs. Driverless cars. Algorithmic journalism. Robots that make hamburgers. Medical machines that read MRI scans. Even, ironically, automated career advice. “Everywhere these students look it seems that jobs are disappearing,” Ken laments.

This is one of the defining anxieties of the developed world. The rapidly changing nature of work is one of the key challenges and opportunities of our time. We need to address the very real and understandable worries people have about the future without denying ourselves the broader benefits of technological change. Future governments will have to deal with a world in which artificial intelligence and automation will creep into every occupation, from bricklayer to teacher. We, in turn, will need to prepare for a working life that even a few years ago was unthinkable.

Humans have a complicated emotional connection with robots – evident in its being a recurrent cultural trope from Mary Shelley’s Frankenstein to Blade Runner and Westworld. We are gripped by a fascination with machines built in our own image, but also by the dread they evoke, the sense they can and will do us harm.

The specific fear that machines will steal our jobs is nearly as old as technology itself. In 1589, Queen Elizabeth I refused to grant the inventor of a mechanical knitting machine, known as a “stocking frame”, a patent for fear of putting hand-knitters out of work. But the idea really became a potent political issue in the 20th century.

In 1930, the British economist John Maynard Keynes warned of “technological unemployment ... due to our discovery of means of economizing the use of labor outrunning the pace at which we can find new uses for labor”. The American economist Wassily Leontief predicted in 1952 that labour would become “less and less important” as new machines made human workers redundant in the same way engine technologies had essentially eliminated the need for horses. In 1935, President Franklin D Roosevelt warned that the US economy might never be able to reabsorb all the workers displaced by the increasing efficiency of machines. Just a few decades later, in the 1960s, President John F Kennedy called it “a major domestic challenge … to maintain full employment at a time when automation … is replacing men”. In 1980, a Time magazine cover titled “The Robot Revolution” showed a tentacled automaton at a conveyor belt, simultaneously welding cars, shearing sheep and creating miniature machines in its own image. And earlier this year Barack Obama warned in his presidential farewell speech that “the next wave of economic dislocation won’t come from overseas. It will come from the relentless pace of automation that makes a lot of good, middle-class jobs obsolete.”

These dire predictions have been right about one thing: technology has destroyed millions of jobs around the world. As foreseen by Queen Elizabeth, the stocking frames, or at least their successors, eventually put “frame breaker” Ned Ludd, who later became a folkloric symbol to “Luddites”, and his fellow weavers out of work. As machines emerged en masse during the industrial revolution, they replaced laundry maids, blacksmiths and thousands of other occupations. Machines in agriculture eliminated more than nine out of every ten farm jobs, with agricultural employment going from being the main occupation of more than 40% of the population at the start of the 20th century to just 2% by its end. In our lifetimes, office technology – photocopiers, fax machines, personal computers, emails, accounting software and cloud computing – has automated many administrative tasks. The secretarial hutches, switchboard desks, typing pools and filing cabinets can occasionally still be found in old office buildings, but the jobs are gone.

Yet the fears of mass unemployment haven’t come to pass. It seems the gloomy soothsayers didn’t overestimate the capability of machines; they underestimated human capacity to change existing jobs and create new ones.

Walk in the door of the new ANZ Bank flagship branch in Sydney’s Martin Place, crane your neck upwards, and you’ll notice the huge digital screen extending three storeys high amid glass and gleaming steel. It all feels less a bank branch than a high-tech cathedral.

Technology is everywhere. The entry vestibule is fitted with a full range of digital banking options. You can use one of the “smart” ATMs to withdraw cash and deposit cash and cheques (although if you’re still using cheques, this store probably isn’t for you). Twenty-one digital screens are laid out around the store, inviting customers to complete banking transactions online. Alternatively, they can use the free wi-fi network to access Apple Pay or ANZ’s own banking app, goMoney. A digital “banker desktop” as well as an iPad “discover bar” provide further information.

But if you think that all this technology has made humans redundant, you’d be wrong. The Martin Place branch still has 15 staff – not many fewer than it’s always had. The role technology has played in branches like this has grown over time. In the 1960s, the development of readable characters led to the first automated reader-sorter machines. Soon after that, emerging computer technology was processing transactions more efficiently than human hands, reducing the need for large administrative teams. Instead of dispensing cash and processing transactions, staff began to extend into increasingly sophisticated and higher-value tasks: approving home loans, selling insurance, giving financial-management advice.

The Martin Place ANZ branch is just one workplace, but it illustrates some important features of the changing relationship between machines and humans.

First, that relationship is not as simple as the “machines will take our jobs” narrative suggests. The automation age is not a zero-sum game, in which machines advance and humans retreat. Rather, machines and humans are racing alongside each other. Machines are improving as technology enables them to perform a growing number of tasks once done by humans (such as with ATMs). But humans are improving, too. We are relentlessly creating new, complex tasks and inventing new jobs (like financial advice and wealth management).

Second, robots can destroy human jobs, but they can also create new ones. In the cold language of economics, machines can “substitute” for workers (for example, when a new ATM reduces the need for bank tellers) but they can also “complement” workers (for example, when that same ATM creates new work for armoured vehicle drivers, repair technicians and other support staff). In fact, fears that employment in retail banks would plummet when ATMs were widely installed in the 1970s and 1980s proved unfounded. Also, US Bureau of Labor Statistics data show that the number of retail bank workers in the United States increased by more than 40% in the three decades since ATMs came into common usage.

Third, and most importantly, the banking example offers us a glimpse into the future – and the jobs we will likely do as technology becomes more capable. Relative to machines, humans still have a comparative advantage in occupations where personal service and human interactions are an important part of the job. This is obvious at the Martin Place bank branch where machines have come to dominate the simpler processing and information tasks on the ground floor, while humans have simply moved upstairs to perform more complex and interpersonal tasks. They are now working on the first floor, helping customers with financial products in specialist appointment rooms. The second floor functions as an open space for seminars and financial training.

For the individuals involved, none of this change is easy, but we have to ask ourselves: Would we want it any other way? If we could click our fingers and recover the tellers counting out notes by hand, would we do it? Or if we could recover the jobs of the comptometrists, who manually checked thousands of additions and subtractions, would we want them back? Would you go back even further and rehire the elevator operators who would have once stood on their feet for six, or eight, or ten hours per day, ferrying customers between floors? Would we want to put the Martin Place lamplighters, street sweepers, clock winders and traffic police back to work?

Career Voyage begins by asking Jay to record how he feels about different work-related activities. The activity on the screen is “Conduct mechanical repairs on a car” and Jay is instructed to check one of five boxes to indicate “strong like”, “like”, “don’t mind”, “dislike” or “strong dislike”. Pausing for a moment, he checks like. The next activity listed is “sketch cartoons” – Jay checks don’t mind. “Repair digital cameras” – dislike. “Organise the planting of a forest” – like.

After Jay answers about a hundred of these questions, Career Voyage spits out a list of jobs based on his preferences: electrical engineer, air traffic controller, mechanic, environmental coordinator and about a dozen others. Jay seems equivocal, which, according to Ken, is not an uncommon response.

“These are just a start,” he says. “Most of these kids will have ten to 15 jobs throughout their lives.” But he says securing a first job for any but the most “switched-on” kids is difficult.

While the new high-tech ANZ branch shows how technology and human work can be mutually reinforcing, the labour market isn’t always so benign. Other features point towards significant challenges ahead.

The first is that workforce change isn’t always smooth, especially for the people affected. The displaced bank tellers weren’t all able to become financial advisers or ATM repair technicians. More broadly, the dynamism of the Australian labour market has forced millions of workers to transition from one job to another. In the past 25 years, Australia has lost nearly 100,000 machinery operator jobs, nearly 400,000 labourers, and nearly 250,000 technicians and trades jobs. Certainly, over the same period, there has been an explosion of more than 400,000 new jobs in community and personal services and 700,000 new jobs across the professional and business services. But the transition for many workers is far from easy. Australia has a poor record of transitioning workers from declining to growing occupations. Over the same 25-year period, nearly one in ten unskilled men who lost their job did not return to the labour force. Today, more than one in four unskilled men don’t participate in the labour market.

The second challenge is that the creation and destruction of jobs aren’t always neatly contemporaneous, so there will be periods of higher and lower unemployment. Many times, the race between humans and machines has seen one or the other pull ahead. In 1940, when giving his State of the Union address, President Roosevelt blamed high joblessness on the nation’s failure to “[find] jobs faster than invention can take them away”. The start of World War Two turned the surplus of labour into a shortage. But in the 1960s, unemployment was rising again, causing President Lyndon B Johnson to convene a national commission to assess the economic effects of automation and technological change. By the time the commission published its report a few years later, the economy was booming once more.

The third challenge is that while technological progress has been consistent with high employment, the same cannot be said for wages. All around us, technological progress is creating value, but too little of that is being shared with workers. Profits have never been higher. Wages growth has never been lower.

For most of the last century, the growing use of machines in workplaces was good for workers’ pay packets. In most advanced economies, workers took home around two thirds of national income, while the owners of the machines pocketed the remaining third. Throughout that time, the number of machines steadily increased relative to the number of workers. Yet the rise of the machines did not cause the workers’ share of income to decline. In fact, workers were able to maintain a stable share of national income. Why? Because the workers’ share of income is equal to the relative number of workers and machines (the ratio of labour to capital) multiplied by their relative payments (the ratio of wages to capital returns). Over the 20th century, these two ratios tended to move in opposite directions – that is, as the number of workers fell relative to machines, the productivity of each worker increased, their wages rose and their share of income stayed constant. This constant labour share was so well established it was described by influential British economist Nicholas Kaldor as one of the “remarkable historical constancies” of economics – a principle that underpinned the social acceptance, even encouragement, of machines into the workplace.

However, workers’ share of national output has been falling over the past three decades. In June this year, The Australia Institute reported that payments to workers (including wages, salaries and super contributions) accounted for just 46.2% of total gross domestic product (GDP): the lowest share in more than 50 years. In Australia and across other developed economies, robots are helping the pie to grow, but workers aren’t getting a larger slice.

Economists debate the reasons for the falling labour share of income. Thomas Piketty, author of the blockbuster tome on global inequality Capital in the Twenty-First Century, argues that a change in the nature of technology is to blame. Piketty believes that, over time, machines have become more autonomous and are now more substitutes than complements for workers. Rather than just providing additional leverage for humans in the workplace, robots can think and learn in ways that replace many human functions. If robots don’t create as much complementary demand for labour, increasing numbers of robots don’t put upward pressure on the wages of workers. Piketty believes this is consistent with what we are seeing now: more machines, lower wages growth.

Other economists put less weight on the changing substitutability between workers and machines. They stress other factors contributing to a falling labour share of output, such as weak bargaining power of labour, political capture, monopolistic behaviour and globalisation. But they all agree that in the modern economy the rising tide doesn’t necessarily lift all boats, and the benefits of technology are becoming harder to distribute.

How do we guard against the risk that technological advances will leave some people behind and the gains of progress will not be fairly shared?

Governments have implemented this type of active policy many times before. There was, for example, essentially no wages growth from the beginning of the industrial revolution, circa 1760, until around 1850, despite its being one of the most intense periods of rapid technological change in history. The first machines developed led to massive increases in production, but it took time for wages to rise. When they finally did, the increase was owed as much to policy and politics as to technology. A new concentration of factory jobs led to a stronger political organisation of workers in Britain. In 1833, a professional factory inspectorate was set up to regulate safe working conditions. In 1846, the “Corn Laws” – tariffs and restrictions on imported food – were repealed, significantly reducing bread prices and raising real wages. The Factory Act 1847 limited working hours in textile mills to ten hours per day for women and teenagers. In 1875, the Master and Servant Acts were repealed, eliminating legally enforceable duties of loyalty and obedience. Gradually, demand for labour started to rise. At the same time, major political changes occurred that distributed political power more widely and helped spread the benefits of technology.

The Luddites’ attempts to cling onto their livelihoods by smashing the weaving frames that had displaced them did little good. What ultimately brought benefits to working people in England was the proliferation of technology, backed by strong political institutions.

The answer then, as now, is confidence in human ability to use technology for the mutual good. There is no economic reason why Australia cannot address today’s labour market inequality and increase employment while also enjoying even greater advances in technology and higher levels of productivity.

Technologies like automation, artificial intelligence and robotics offer the opportunity to reimagine what it means to work. We should approach this task with a sense of optimism. The chance to reinvent what we mean by work represents a remarkable opportunity for humanity. For 10,000 years, since the invention of agriculture, work for most people has meant difficult physical tasks: lifting, digging, bashing, hauling, scrubbing. In the last century, the fastest growing occupations involved less manual work and more administrative tasks: typing, filing, copying, serving, selling and driving. In their time, the jobs comprised of these manual and routine tasks provided dignified, valuable work to our ancestors. But it shouldn’t diminish the dignity or value of that work to say that our generation is on the cusp of something different.

By redefining how we want to work, we can create new jobs for a future in which humans and machines can complement each other. While the robots toil away on the routine and physical tasks, humans can focus on interpersonal, creative and problem-solving tasks. To enable this shift, we will need to re-evaluate what we mean by “meaningful work” and expand the definition of “job” to include many more activities that deliver value to society but are currently not thought of under that rubric. The question we will need to answer is what makes us uniquely human. In the future, more of us will work with our brains, rather than our hands.

We will need to embrace technology and boost skills. These are the raw ingredients of new job creation. Currently, Australia is falling behind – only 9% of our large companies are embracing automation. A change in mindset is needed, so that politicians, businesses and the community see robots, artificial intelligence and other technologies not as a threat but as a force that can be harnessed for good. To support these new jobs, our education system also needs a massive overhaul. At present, more than two-thirds of Australian students in universities and vocational education are enrolled in courses that seek to prepare them for jobs that will most likely disappear in coming years. There is a huge gap between the skills Australia’s young people learn at school and university and the ones they’ll actually need to succeed in the workplaces of the near future.

Constant, unpredictable change in the demand for skills will be an unavoidable feature of tomorrow’s labour market. We cannot pretend that the transition from old to new jobs will be seamless, that workers will be able to trade their hard hats for health care jobs easily. New labour market approaches will be needed that help workers adjust to the new reality, including job training, skills-matching programs and relocation assistance.

Finally, we will need to ensure fairness. In a fast-changing labour market, strong institutions will safeguard conditions and ensure that productivity gains are distributed fairly to workers. These institutions will need to be supported by organised workers who can maintain a fair balance of power in the workplace. As the historian Colin Gordon has observed, labour’s share of income has fallen most rapidly in those sectors where “union presence withered, not where computers displaced labor”.

Company leaders and workers should find common cause in their response to technology. Ultimately, despite sitting on opposite ends of the wage-bargaining table, neither side benefits if the other gets too far ahead. A story of leading American unionist Walter Reuther has it that Henry Ford II, grandson of the famous automaker’s founder, once asked him during a visit at a newly built and highly automated car factory: “How are you going to get those robots to pay your union dues?” To which Reuther replied: “Henry, how are you going to get them to buy your cars?”

The current fashion for a deterministic view of history (human and machine labour in conflict not cooperation) and technological pessimism (all human labour will become inferior) is leading to policies to try to hold back the tide of automation. For others, it’s causing them to give up on the idea of meaningful human work altogether and prepare large parts of the population to lead workless lives.

One proposal motivated by this pessimism and the perceived threat of mass unemployment is a universal basic income (UBI). It springs from a belief that the inevitable consequence of technological progress is that most human workers will be sitting around idle. In fact, many tech entrepreneurs in Silicon Valley and beyond appear so optimistic about the supremacy of their emerging technologies that they see no other solution than to put the rest of humanity on gardening leave with a guaranteed monthly cheque of a few thousand dollars. In recent months, Mark Zuckerberg, Elon Musk and Richard Branson have all publicly talked about the desirability of a UBI. From their vantage point, it may indeed seem like a socially responsible way to cushion the unemployment effect of automation and reduce poverty.

However, a UBI would inevitably worsen inequality. Moving from the welfare system we have today to one handing out large sums to the already well off would be inherently inequitable. Some UBI proponents have suggested coupling the model with a progressive income tax, but that raises the question: why not keep the tax and leave the UBI?

A UBI also misses the great opportunity in the changing labour market. And because joblessness is associated with worse health outcomes, rising crime rates and increased substance abuse, passively accepting the tale of technology-driven mass unemployment that can only be fixed with a UBI would also put social stability at risk. Rather than closing the door on human work, technology offers the opportunity to continuously reinvent work and find new jobs. The lesson from history is that we should reject policies that take a less optimistic view of human potential.

It doesn’t take much to unsettle a teenager in the final years of high school. There is the family pressure to do well. The talk about house prices being out of reach for their generation. And the assertions that an army of robots is waiting to take over half of all human jobs in coming years don’t help.

“These kids are used to people hammering down negative messages. It does affect their outlook,” Ken Tinnet says. Every now and again an industry expert does the rounds at the region’s schools to talk about career prospects. “Many presenters just put up their statistics and say, ‘It’s going to be tough’, ‘The labour market is tough’,” says Ken, who started as the school’s head teacher for vocational training almost two decades ago, and can’t remember if things used to be easier back then.

So, when one careers expert walked into the school’s auditorium earlier this year and told the fidgety student crowd not to worry because there would be lots of jobs in the future, Ken was pleasantly surprised. “That was reassuring,” he says. “It was a message of hope.”