In early 1984, executives at the telephone company Pacific Bell made a fateful decision. For decades, the company had enjoyed a virtual monopoly on telephone services in California, but now it was facing a problem. The industry was about to be deregulated, and Pacific Bell would soon be facing tough competition.

The management team responded by doing all the things managers usually do: restructuring, downsizing, rebranding. But for the company executives, this wasn’t enough. They worried that Pacific Bell didn’t have the right culture, that employees did not understand “the profit concept” and were not sufficiently entrepreneurial. If they were to compete in this new world, it was not just their balance sheet that needed an overhaul, the executives decided. Their 23,000 employees needed to be overhauled as well.

The company turned to a well-known organisational development specialist, Charles Krone, who set about designing a management-training programme to transform the way people thought, talked and behaved. The programme was based on the ideas of the 20th-century Russian mystic George Gurdjieff. According to Gurdjieff, most of us spend our days mired in “waking sleep”, and it is only by shedding ingrained habits of thinking that we can liberate our inner potential. Gurdjieff’s mystical ideas originally appealed to members of the modernist avant garde, such as the writer Katherine Mansfield and the architect Frank Lloyd Wright. More than 60 years later, senior executives at Pacific Bell were likewise seduced by Gurdjieff’s ideas. The company planned to spend $147m (£111m) putting their employees through the new training programme, which came to be known as Kroning.

Over the course of 10 two-day sessions, staff were instructed in new concepts, such as “the law of three” (a “thinking framework that helps us identify the quality of mental energy we have”), and discovered the importance of “alignment”, “intentionality” and “end-state visions”. This new vocabulary was designed to awake employees from their bureaucratic doze and open their eyes to a new higher-level consciousness. And some did indeed feel like their ability to get things done had improved.

But there were some unfortunate side-effects of this heightened corporate consciousness. First, according to one former middle manager, it was virtually impossible for anyone outside the company to understand this new language the employees were speaking. Second, the manager said, the new language “led to a lot more meetings” and the sheer amount of time wasted nurturing their newfound states of higher consciousness meant that “everything took twice as long”. “If the energy that had been put into Kroning had been put to the business at hand, we all would have gotten a lot more done,” said the manager.

Although Kroning was packaged in the new-age language of psychic liberation, it was backed by all the threats of an authoritarian corporation. Many employees felt they were under undue pressure to buy into Kroning. For instance, one manager was summoned to her superior’s office after a team member walked out of a Kroning session. She was asked to “force out or retire” the rebellious employee.

Some Pacific Bell employees wrote to their congressmen about Kroning. Newspapers ran damning stories with headlines such as “Phone company dabbles in mysticism”. The Californian utility regulator launched a public inquiry, and eventually closed the training course, but not before $40m dollars had been spent.

During this period, a young computer programmer at Pacific Bell was spending his spare time drawing a cartoon that mercilessly mocked the management-speak that had invaded his workplace. The cartoon featured a hapless office drone, his disaffected colleagues, his evil boss and an even more evil management consultant. It was a hit, and the comic strip was syndicated in newspapers across the world. The programmer’s name was Scott Adams, and the series he created was Dilbert. You can still find these images pinned up in thousands of office cubicles around the world today.

Although Kroning may have been killed off, Kronese has lived on. The indecipherable management-speak of which Charles Krone was an early proponent seems to have infected the entire world. These days, Krone’s gobbledygook seems relatively benign compared to much of the vacuous language circulating in the emails and meeting rooms of corporations, government agencies and NGOs. Words like “intentionality” sound quite sensible when compared to “ideation”, “imagineering”, and “inboxing” – the sort of management-speak used to talk about everything from educating children to running nuclear power plants. This language has become a kind of organisational lingua franca, used by middle managers in the same way that freemasons use secret handshakes – to indicate their membership and status. It echoes across the cubicled landscape. It seems to be everywhere, and refer to anything, and nothing.

It hasn’t always been this way. A certain amount of empty talk is unavoidable when humans gather together in large groups, but the kind of bullshit through which we all have to wade every day is a remarkably recent creation. To understand why, we have to look at how management fashions have changed over the past century or so.

In the late 18th century, firms were owned and operated by businesspeople who tended to rely on tradition and instinct to manage their employees. Over the next century, as factories became more common, a new figure appeared: the manager. This new class of boss faced a big problem, albeit one familiar to many people who occupy new positions: they were not taken seriously. To gain respect, managers assumed the trappings of established professions such as doctors and lawyers. They were particularly keen to be seen as a new kind of engineer, so they appropriated the stopwatches and rulers used by them. In the process, they created the first major workplace fashion: scientific management.

Charlie Chaplin ‘satirising the cult of scientific management’ in 1936 film Modern Times. Photograph: Allstar/Cinetext

Firms started recruiting efficiency experts to conduct time-and-motion studies. After recording every single movement of a worker in minute detail, the time-and-motion expert would rearrange the worker’s performance of tasks into a more efficient order. Their aim was to make the worker into a well-functioning machine, doing each part of the job in the most efficient way. Scientific management was not limited to the workplaces of the capitalist west – Stalin pushed for similar techniques to be imposed in factories throughout the Soviet Union.

Workers found the new techniques alien, and a backlash inevitably followed. Charlie Chaplin famously satirised the cult of scientific management in his 1936 film Modern Times, which depicts a factory worker who is slowly driven mad by the pressures of life on the production line.

As scientific management became increasingly unpopular, executives began casting around for alternatives. They found inspiration in a famous series of experiments conducted by psychologists in the 1920s at the Hawthorne Works, a factory complex in Illinois where tens of thousands of workers were employed by Western Electric to make telephone equipment. A team of researchers from Harvard had initially set out to discover whether changes in environment, such as adjusting the lighting or temperature, could influence how much workers produced each day.

To their surprise, the researchers found that no matter how light or dark the workplace was, employees continued to work hard. The only thing that seemed to make a difference was the amount of attention that workers got from the experimenters. This insight led one of the researchers, an Australian psychologist called Elton Mayo, to conclude that what he called the “human aspects” of work were far more important than “environmental” factors. While this may seem obvious, it came as news to many executives at the time.

As Mayo’s ideas caught hold, companies attempted to humanise their workplaces. They began talking about human relationships, worker motivation and group dynamics. They started conducting personality testing and running teambuilding exercises: all in the hope of nurturing good human relations in the workplace.

This newfound interest in the human side of work did not last long. During the second world war, as the US and UK military invested heavily in trying to make war more efficient, management fashions began to shift. A bright young Berkeley graduate called Robert McNamara led a US army air forces team that used statistics to plan the most cost-effective way to flatten Japan in bombing campaigns. After the war, many military leaders brought these new techniques into the corporate world. McNamara, for instance, joined the Ford Motor Company, rising quickly to become its CEO, while the mathematical procedures that he had developed during the war were enthusiastically taken up by companies to help plan the best way to deliver cheese, toothpaste and Barbie dolls to American consumers. Today these techniques are known as supply-chain management.

During the postwar years, the individual worker once again became a cog in a large, hierarchical machine. While many of the grey-suited employees at these firms savoured the security, freedom and increasing affluence that their work brought, many also complained about the deep lack of meaning in their lives. The backlash came in the late 1960s, as the youth movement railed against the conformity demanded by big corporations. Protesters sprayed slogans such as “live without dead time” and “to hell with boundaries” on to city walls around the world. They wanted to be themselves, express who they really were, and not have to obey “the Man”.

In response to this cultural change, in the 1970s, management fashions changed again. Executives began attending new-age workshops to help them “self actualise” by unlocking their hidden “human potential”. Companies instigated “encounter groups”, in which employees could explore their deeper inner emotions. Offices were redesigned to look more like university campuses than factories.

Mad Men’s liberated adman Don Draper (Jon Hamm). Photograph: Courtesy of AMC/AMC

Nowhere is this shift better captured than in the final episode of the television series Mad Men. Don Draper had been the exemplar of the organisational man, wearing a standard-issue grey suit when we met him at the beginning of the show’s first series. After suffering numerous breakdowns over the intervening years, he finds himself at the Esalen institute in northern California, the home of the human potential movement. Initially, Draper resists. But soon he is sitting in a confessional circle, sobbing as he tells his story. His personal breakthrough leads him to take up meditating and chanting, looking out over the Pacific Ocean. The result of Don Draper’s visit to Esalen isn’t just personal transformation. The final scene shows the now-liberated adman’s new creation – an iconic Coca-Cola commercial in which a multiracial group of children stand on a hilltop singing about how they would like to buy the world a Coke and drink it in perfect harmony.

After the fictional Don Draper visited Esalen, work became a place you could go to find yourself. Corporate mission statements now sounded like the revolutionary graffiti of the 1960s. The company training programme run by Charles Krone at Pacific Bell came straight from the Esalen playbook.

Since new-age ideas first permeated the workplace in the 1970s, the spin cycle of management-speak has sped up. During the 1980s, management experts went in search of fresh ideas in Japan. Management became a kind of martial art, with executives visiting “quality dojos” to earn “lean black-belts”. In their 1982 bestseller, In Search of Excellence, Tom Peters and Robert Waterman – both employees of McKinsey, the huge management consultancy agency – recommended that firms foster the same commitment to the company that they found among Honda employees in Japan. The book included the story of one Japanese employee who happens upon a damaged Honda on a public street. He stops and immediately begins repairing the car. The reason? He can’t bear to see a Honda that isn’t perfect.

While McKinsey consultants were mining the wisdom of the east, the ideas of Harvard Business School’s Michael Jensen started to find favour among Wall Street financiers. Jensen saw the corporation as a portfolio of assets. Even people – labelled as “human resources” – were part of this portfolio. Each company existed to create returns for shareholders, and if managers failed to do this, they should be fired. If a company didn’t generate adequate returns, it should be broken up and sold off. Every little part of the company was seen as a business. Seduced by this view, many organisations started creating “internal markets”. In the 1990s, under director general John Birt, the BBC created a system in which everything from time in a recording studio to toilet cleaning was traded on a complex internal market. The number of accountants working for the broadcaster exploded, while people who created TV and radio shows were laid off.

As companies have become increasingly ravenous for the latest management fad, they have also become less discerning. Some bizarre recent trends include equine-assisted coaching (“You can lead people, but can you lead a horse?”) and rage rooms (a room where employees can go to take out their frustrations by smashing up office furniture, computers and images of their boss).

A century of management fads has created workplaces that are full of empty words and equally empty rituals. We have to live with the consequences of this history every day. Consider a meeting I recently attended. During the course of an hour, I recorded 64 different nuggets of corporate claptrap. They included familiar favourites such as “doing a deep dive”, “reaching out”, and “thought leadership”. There were also some new ones I hadn’t heard before: people with “protected characteristics” (anyone who wasn’t a white straight guy), “the aha effect” (realising something), “getting our friends in the tent” (getting support from others).

After the meeting, I found myself wondering why otherwise smart people so easily slipped into this kind of business bullshit. How had this obfuscatory way of speaking become so successful? There are a number of familiar and credible explanations. People use management-speak to give the impression of expertise. The inherent vagueness of this language also helps us dodge tough questions. Then there is the simple fact that even if business bullshit annoys many people, in most work situations we try our hardest to be polite and avoid confrontation. So instead of causing a scene by questioning the bullshit flying around the room, I followed the example of Simon Harwood, the director of strategic governance in the BBC’s self-satirising TV sitcom W1A. I used his standard response to any idea – no matter how absurd – “hurrah”.

Still, these explanations did not seem to fully account for the conquest of bullshit. I came across one further explanation in a short article by the anthropologist David Graeber. As factories producing goods in the west have been dismantled, and their work outsourced or replaced with automation, large parts of western economies have been left with little to do. In the 1970s, some sociologists worried that this would lead to a world in which people would need to find new ways to fill their time. The great tragedy for many is that just the opposite seems to have happened.

Simon Harwood (Jason Watkins, centre) of W1A, the BBC’s fictional director of strategic governance. Photograph: Jack Barnes/BBC

At the very point when work seemed to be withering away, we all became obsessed with it. To be a good citizen, you need to be a productive citizen. There is only one problem, of course: there is less than ever that actually needs to be produced. As Graeber pointed out, the answer has come in the form of what he calls “bullshit jobs”. These are jobs in which people experience their work as “utterly meaningless, contributing nothing to the world”. In a YouGov poll conducted in 2015, 37% of respondents in the UK said their job made no meaningful contribution to the world. But people working in bullshit jobs need to do something. And that something is usually the production, distribution and consumption of bullshit. According to a 2014 survey by the polling agency Harris, the average US employee now spends 45% of their working day doing their real job. The other 55% is spent doing things such as wading through endless emails or attending pointless meetings. Many employees have extended their working day so they can stay late to do their “real work”.

One thing continued to puzzle me: why was it that so many people were paid to do this kind of empty work. One reason that David Graeber gives, in his book The Utopia of Rules, is rampant bureaucracy: there are more forms to be filled in, procedures to be followed and standards to be complied with than ever. Today, bureaucracy comes cloaked in the language of change. Organisations are full of people whose job is to create change for no real reason.

Manufacturing hollow change requires a constant supply of new management fads and fashions. Fortunately, there is a massive industry of business bullshit merchants who are quite happy to supply it. For each new change, new bullshit is needed. Looking back over the list of business bullshit I had noted down during the meeting, I realised that much of it was directly related to empty new bureaucratic initiatives, which were seen as terribly urgent, but would probably be forgotten about in a few years’ time.

One of the corrosive effects of business bullshit can be seen in the statistic that 43% of all teachers in England are considering quitting in the next five years. The most frequently cited reasons are increasingly heavy workloads caused by excessive administration, and a lack of time and space to devote to educating students. A remarkably similar picture appears if you look at the healthcare sector: in the UK, 81% of senior doctors say they are considering retiring from their job early; 57% of GPs are considering leaving the profession; 66% of nurses say they would quit if they could. In each case, the most frequently cited reason is stress caused by increasing managerial demands, and lack of time to do their job properly.

It is not just employees who feel overwhelmed. During the 1980s, when Kroning was in full swing, empty management-speak was confined to the beige meeting rooms of large corporations. Now, it has seeped into every aspect of life. Politicians use business balderdash to avoid grappling with important issues. The machinery of state has also come down with the word-virus. The NHS is crawling with “quality sensei”, “lean ninjas”, and “blue-sky thinkers”. Even schools are flooded with the latest business buzzwords like “grit”, “flipped learning” and “mastery”. Naturally, the kids are learning fast. One teacher recalled how a seven-year-old described her day at school: “Well, when we get to class, we get out our books and start on our non-negotiables.”

In the introduction to his 2015 book, Trust Me, PR Is Dead, the former PR executive Robert Phillips tells a fascinating story. One day he was called up by the CEO of a global corporation. The CEO was worried. A factory which was part of his firm’s supply chain had caught fire and 100 women had burned to death. “My chairman’s been giving me grief,” said the CEO. “He thinks we’re failing to get our message across. We are not emphasising our CSR [corporate social responsibility] credentials well enough.” Phillips responded: “While 100 women’s bodies are still smouldering?” The CEO was “struggling to contain both incredulity and temper”. “I know,” he said. “Please help.” Phillips responded: “You start with actions, not words.”

In many ways, this one interaction tells us how bullshit is used in corporate life. Individual executives facing a problem know that turning to bullshit is probably not the best idea. However, they feel compelled. The problem is that such compulsions often cloud people’s best judgements. They start to think empty words will trump reasonable reflection and considered action. Sadly, in many contexts, empty words win out.

If we hope to improve organisational life – and the wider impact that organisations have on our society – then a good place to start is by reducing the amount of bullshit our organisations produce. Business bullshit allows us to blather on without saying anything. It empties out language and makes us less able to think clearly and soberly about the real issues. As we find our words become increasingly meaningless, we begin to feel a sense of powerlessness. We start to feel there is little we can do apart from play along, benefit from the game and have the occasional laugh.

But this does not need to be the case. Business bullshit can and should be challenged. This is a task each of us can take up by refusing to use empty management-speak. We can stop ourselves from being one more conduit in its circulation. Instead of just rolling our eyes and checking our emails, we should demand something more meaningful.

Clearly, our own individual efforts are not enough. Putting management-speak in its place is going to require a collective effort. What we need is an anti-bullshit movement. It would be made up of people from all walks of life who are dedicated to rooting out empty language. It would question management twaddle in government, in popular culture, in the private sector, in education and in our private lives.

The aim would not just be bullshit-spotting. It would also be a way of reminding people that each of our institutions has its own language and rich set of traditions which are being undermined by the spread of the empty management-speak. It would try to remind people of the power which speech and ideas can have when they are not suffocated with bullshit. By cleaning out the bullshit, it might become possible to have much better functioning organisations and institutions and richer and fulfilling lives.

Business Bullshit by André Spicer is published by Taylor & Francis. Buy it for £16.99 at bookshop.theguardian.com

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.