Fiona Stanley was late. Rushing through the corridors of a children's hospital in search of a meeting, the much celebrated epidemiologist, research scientist and 2003 Australian of the Year burst through a door. But instead of a ward or an office with people hard at work, what she saw astounded her. She exclaims now, 10 years later, "It was the entire senior team of the hospital, seated in pairs, holding hands and looking deeply into each other's eyes. All I could think was, 'What on earth are they doing? And why aren't they running the hospital and making it great?'" There are other questions. How had such a group been persuaded to take part? Who had done that so effectively? What did it cost? Stanley is incandescent about some of the boards and management teams she sees running some of our hospitals. "We need science and evidence to guide decisions, not this PR branding and management claptrap!" she exclaims. Dennis Muilenburg is the 55-year-old chair, president and CEO of Boeing. In mid-March, he issued an open letter to passengers, airlines and the industry, which was also published in Australian newspapers. It was titled: "Our Commitment", and followed two crashes within five months which killed 346 people. Both had involved the new Boeing 737 Max 8. Preliminary findings into the first crash implicated a flight control software change that Boeing had not told its airline customers about.

Instead, it had marketed the planes, its bestselling model so far, to cost-conscious airlines as needing minimal training for pilots already used to flying the Boeing 737. When recovered, the black box from that crash, on Lion Air flight 610, revealed the pilots frantically flipping through the Boeing manual trying to work out how to stop the plane's repeated nose-diving. The same anti-stall software was then implicated in the second crash. US federal prosecutors are now investigating. After the second crash, the US became one of the last countries to ground the planes. Almost immediately, reporters revealed such a close relationship between Boeing and the Federal Aviation Authority that Boeing's own people had virtually done the FAA safety certification of the new aircraft. Who knew that? Not the average Boeing passenger. In his letter, Muilenburg told the public: "Our purpose at Boeing is to bring family, friends and loved ones together with our commercial aircraft – safely." The 2018 Banking royal commission was shockingly rich in excruciating moments for people more used to being either invisible, except to their peers, or celebrated in the business press. Here's one cameo. It is August 8 and Nicole Smith, the former chair of NAB's superannuation trustee NULIS Nominees, is giving evidence about financial advice fees charged to the bank's superannuation customers. Counsel assisting Michael Hodge, QC, has been quizzing Smith about the bank's practice of not assigning an adviser to some customers – but then charging the fees regardless. Commissioner Kenneth Hayne intervened, asking Smith: "Did you think yourself that taking money to which there was no entitlement raised a question for criminal law?"

"I didn't," Smith replied. Smith has not been charged with any criminal offences. A recent New Yorker cartoon shows a man, surrounded by puffy clouds and standing at the entrance to Heaven. In front of him is St Peter, and above them, topping the curly wrought-iron gates, is a bright red neon sign: BIRTH, DEATH & BEYOND. St Peter says gloomily, "Actually, I preferred 'Heaven', too, but then the marketing guys got a hold of it." It has taken us a long time to get here, to this slippery, surprising world of ours. Blurriness is everywhere. Each day, just by absorbing the news headlines, or turning on our devices and opening our email or hopping onto Facebook, Google, Twitter, LinkedIn or Instagram, we find ourselves navigating a shifting landscape of spin, sham, fake news, false claims, phishing, pretence, exaggeration, obfuscation, contradiction, empty promises, extravagant PR and outright lies, scams and fraud. And that's before we head into work. Headhunter Ian Smith, of Blenheim Partners, musing on what came out of the banking royal commission, says, "I don't think people are becoming fundamentally more dishonest but there are more opportunities to be so." For yes, deception has always been with us – where would Shakespeare have been without it? – but the rapid social, technological, political and economic shifts of the past several decades have combined to send us rushing down tricky paths. Tricky is the new normal. We can't say we weren't warned.

In 2004, well before the internet's innovations were amplifying every trend at warp speed, British author Francis Wheen had a best seller with How Mumbo-Jumbo Conquered the World. Wheen argued we were seeing the collapse of reason and the rise of "lucrative twaddle", including management consultant jargon, globalisation evangelism, "woo-woo" science, post-modern mantras, New Age mysticism, business-world guru-dom, economic fundamentalists, and pop psychology. All of it designed, Wheen wrote, to get in the way of truth and rationality and "consign us all to a life in darkness". Wheen was just one of many throwing up their hands in public and finding a ready audience in the rest of us. If it weren't for the fact that we have ended up somewhere that, depending on your politics and the size of your bank account and power, is either utterly horrifying or ripe with profitable opportunities, it would feel like satire. Certainly, on our journey here, many writers have resorted to humour to sell the serious message that, dear God, we seem to have gone off-piste. My personal favourites were two laconic authors, also British, who published Is It Just Me or Is Everything Shit? in 2005. Steve Lowe and Alan McArthur took bullseye potshots at all the flummery unfolding in front of them. Here they are on the predilection of the famous, celebrated and pampered, from Martha Stewart to Johnny Depp, to liken themselves to Nelson Mandela: "In a major interview … Bill Clinton revealed that what got him through the Lewinsky scandal and Starr inquiry was the example of Nelson Mandela – who had been in pretty much the same situation … Mandela got into his spot of trouble by fighting one of the most powerful and effective systems of oppression ever conceived, apartheid. Clinton got into his spot of trouble spunking on a young woman's dress. Mandela was jailed for 27 years. Clinton was told off a bit." In Clinton's self-aggrandising, self-pitying analogy, we have one of the factors that has led us into our current predicament: grandiosity, fed by unashamed self-interest, oversized confidence and a seeming inability to recognise nonsense. Or honesty. It's grandiosity, and its darker sides, that lets people take the self-serving, the self-promoting, the far-fetched, the preposterous and the basest of motives, wrap this all up in different wording, give it a good polish and a thick cloak of elite, sometimes hip, respectability, and present it to the world – that is, to rest of us – as the aspirational behaviour, attitudes and outcomes that are to be prized in today's fast-changing, "agile", slick, winners-get-it-all society. Many writers have resorted to humour to sell the serious message that, dear God, we seem to have gone off -piste.

This could also be called the age of fudging and fudgers, of masqueraders and the kind of moves that former Labor powerbroker Graham Richardson once described as "whatever it takes". Along the way, there are all the scandals, mismanagement, tragedies and botches that can flourish when a "whatever it takes" society loses sight of what truth and trust actually are. Entrepreneur Elizabeth Holmes raised $US900m before being indicted for fraud and conspiracy. Credit:AAP One of the the most astonishing stories of deception of recent times is that of Elizabeth Holmes, a Stanford University dropout who raised $US900 million for a Silicon Valley company, Theranos, based on shoddy blood-testing science that ended up risking the lives of both patients and healthy people. It's hard not to be repelled by the pathological lying and chilly narcissism of Holmes, whose company, at its peak in 2014, was valued at $US9 billion. At one stage, she was personally worth $US4.5 billion before she was indicted in mid-2018 for fraud and conspiracy. But almost as repellent is the roll-call of powerful and wealthy people, especially older men, who, apparently blinded by Holmes's looks, fake science and confidence, took board positions, invested in Theranos, or both, often, according to news reports, without audited financial statements. Those names include chemical engineer Channing Robertson, her former Stanford professor, former US secretaries of state Henry Kissinger and George P. Shultz, Oracle founder Larry Ellison and Rupert Murdoch, who alone invested $US125 million. Given how tough life is getting for the average person trying to make an honest living, there is something profoundly unsettling about that scenario. Really? It's that easy to be a con artist now?

Abby Ellin is a canny and successful New York-based journalist who, in one five-year period, found herself deceived not by one man, but by two. In a row. When she discovered the truth about the second – he was married and a skilled, opportunistic liar; the first had been a fantasist who asked her to marry him after pretending to be a US Navy SEAL and a secret agent – she says she wanted to stab herself in the head with a fork. "It is the most frightening thing we can imagine," says Ellin, via phone from Manhattan, "people who aren't what they seem." "I don't think there are more dishonest people," she insists, echoing headhunter Ian Smith, "but there are many more ways to be dishonest. It's technology that has done it." She went on to write a book about deception, Duped – Compulsive Liars and How They Can Deceive You. It is full of heart-rending research, such as the fact that of 1300 people defrauded in romantic scams, 90 per cent of them admitted a third party had tried to warn them. But Ellin also wants her book to be read as a broader investigation. The finding that we don't listen to warnings about a romantic scammer because "we protect ourselves from information that's too painful to digest, especially when we have a lot invested in it being true" has wider implications. It could apply to a politician we want to keep believing, a boss we admire, a financial adviser promising fast returns, or a friend we want to keep trusting. "The line between delusions and dreams is flimsy," she writes. Loading

In conversation, Ellin is funny, sharp and warm. As part of her digging, she wanted to find out why she had been stung twice. Was she needy, vulnerable? No – like so many other men and women, she exhibited a suite of qualities that made her successful in her work and life, but which made her catnip to predatory manipulators. She had high levels of conscientiousness which made her dependable, self-disciplined and achievement-oriented. She was open, warm and approachable. She has since learnt to be wary about everyone and every business, even about the safety claims behind the diet soft drink she glugs every day. "Everybody is working an agenda. Look at Wall Street. There's such an incentive to make money. You feel like a loser if you don't." But she adds, "Everybody's a liar these days. The culture is forcing us to brag, and when you brag, you're going to have to fudge a bit. I might look fabulous on social media but no one knows my stomach is hurting. Everybody is obfuscating; there's so much self-promotion. Then we expect to be congratulated because we put up a picture of ourselves with no filter." One human quality, and the praise now heaped on it, has helped get us here – confidence. It's rare to find a consultant, a politician, a C-suite executive, a TED-talker, a venture capitalist or, of course, a fraudster, who is short of it. Billionaire entrepreneur Richard Branson says it is the most important quality a leader can have. And yet. Almost 25 years ago, David Dunning, a psychology professor at Cornell University in upper New York state, became fascinated by a trait he had noticed in some of his students taking tests: "They expressed all sorts of confidence about how they'd done but no, they hadn't done all that well."

He wondered about its relevance in the workplace: "I was trying to figure out: do incompetent people really not know how badly they are performing?" On the phone from Ann Arbor, where Dunning now works at the University of Michigan, he marvels as he recalls how he and a graduate student, Justin Kruger, "decided to take a look at the people who were doing really poorly and we tested them on logic, grammar and humour. And what we discovered was that – after the tests and after they had seen the responses of other, more competent students – the people at the bottom wouldn't revise their self-impression at all." They continued to over-estimate how well they had done. "After that, even I was convinced the theory was right," Dunning laughs. Their 1999 findings are known as the Dunning-Kruger effect. It explains how many incompetent people not only are confident that they are competent but, it also turns out, when they see real competence, their incompetence means they can't recognise it. Meanwhile, in a true catch-22, competent people, because they know what they don't know, will underestimate their competence – and often be less confident. Dunning and Kruger's findings spawned hundreds of articles and features, and a musical rendition, The Incompetence Opera. In 2000, the authors were awarded the Ig Nobel prize, which celebrates research "that makes you laugh, then think". The election of uber-confident businessman Donald Trump to the White House in 2016 only gave the theory fresh life. "We do like confident people," Dunning says. "We take confidence as evidence that the person really knows what they're talking about … but there are a lot of areas where we know that isn't necessarily true. Confident eyewitnesses, for example." Then there's what else confidence may be covering up. Tomas Chamorro-Premuzic, a professor of business psychology at University College London and at Columbia University, has written an entire book about the human error of mistaking confidence for competence. "Traits like overconfidence and self-absorption should be seen as red flags," he writes. "But instead they prompt us to say, 'Ah, there's a charismatic fellow! He's probably leadership material.'" When I phone him in New York, he says, "If you're looking at a dentist or a doctor or pilot, you don't care if they're confident. This could also be called the age of fudging and fudgers.

It's – 'Do they know their stuff?'" But a different rule seems to apply in business, where the line between confidence and overconfidence can blur. In his new book, Why Do So Many Incompetent Men Become Leaders? (And How To Fix It), he writes, "People tend to equate leadership with the very behaviours – overconfidence for example – that often signal bad leadership." He points out that the world's leadership success rate is woeful. A global 2011 study he cites is so bad I laugh: "More than 14,000 human resources professionals and other managers rated barely 26 per cent of their current leaders positively." Narcissism takes its toll. Simon McKeon, Monash University chancellor and a former chairman of CSIRO and AMP, comments with a corporate heavyweight's insider knowledge: "There are pockets in Australia where there is a tendency to focus on self. The issue of self-preservation can be very, very strong. The more one is concerned about that, the less capacity one has to promote the objectives of the organisation being served, let alone some greater good. The temptation to promote oneself is high, particularly the further up the ladder you get." Over the past 16 years, Richard Whitington, who recently retired as a headhunter in Sydney, has noted the rise of the leadership cult in business – and the growing internal corporate communications teams installed to feed the cult. "Increasingly, CEOs are directing a lot of budget into their own internal reputation building. All of these CEOs are super-scripted and rehearsed," he says. He comments on one former high flyer who crashed after leadership lapses led to scandal: "For the 'corp comms' team, he was 'great talent'. You could wind him up and he'd perform brilliantly. But he was a classic case of a guy who couldn't cope when things went off script." It's hard to read the daily news and not be staggered by the numerous stories of incompetence and worse in highly paid places. A discussion with a lecturer in management leads me to a startling 38-second video clip. He and I had been talking about the seemingly unstoppable growth of management consultancies, outside contractors and professional services firms. (Well, now that so many full-time employees have been let go in both the private and public sectors, often on the advice of outside consultants, someone has to do the work.) The February 2018 clip is from a British parliamentary inquiry and shows British Labour MP Peter Kyle grilling two hapless senior audit partners from the giant professional services firm KPMG. He finally snaps: "I wouldn't hire you to do an audit of the contents of my fridge." The inquiry was examining how the massive government contractor Carillion could have gone into liquidation, with liabilities of £7 billion, when its accounts had been signed off months earlier by KPMG. Indeed, all four of the big accounting firms – KPMG, Deloitte, PwC and EY – were implicated in the disaster, having together earned close to £72 million over 10 years for auditing and advisory work for the company.

The clip is both ghastly and funny, and cathartic for anyone who has watched years of fiascos involving big business, management consultancies, contractors and advisory firms, ranging from the $US74 billion implosion of McKinsey darling Enron in 2001, to Australia's 2016 IT Census failure, to what we learnt about our finance industry's behaviour via the banking royal commission. (And yet, some headlines: "Bank, AMP shares surge on royal commission findings"; "KPMG partners receive bumper payments despite Carillion fallout".) It's vital to remember that in most instances of what looks like bone-headed idiocy, poor or deceptive conduct or even, self-deception, someone somewhere is actually making money. Staggering fortunes have been made in the globalised and digital tech era and we – or at least the media, especially, the business press – have become fixated on Rich Lists and statistics about the rich. (According to Oxfam, the world's richest 26 people now own as much wealth as the poorest half.) At the same time, an approach which Harvard Business School professor Michael Porter has called the "shareholder-value mindset" swept through the private and then the public sector after the deregulatory and reform fervour of the early 1980s. It brought an intense focus on cutting costs and jobs while pushing profits towards shareholders and management. Longer-term aims – planned growth, eventual good – became less important. Fiona Stanley calls it "the MBA-from-Harvard syndrome" and is horrified by its application to hospitals, universities and the like: "And once you got a chairman who was the MBA-from-Harvard chairman, you were doomed. Somehow, the reason you're actually existing went out the window." Nothing concentrates the mind more on money than not having enough of it, especially when you see so many others – CEOs, C-suiters, financiers, management consultants – making it hand over fist, turning themselves into what many began to mutter was a financial elite increasingly distant from the rest of us. Rod Sims, head of the Australian Competition and Consumer Commission, finally put that view on the front page of the Australian Financial Review, telling one reporter in January, "I think there's an element of arrogance about corporate Australia, that they are in a privileged position and they can do as they like in almost an unfettered way." But the obsession with money, status and privilege continues. In mid-March, who wasn't gobsmacked to see a bunch of Hollywood stars and powerful American executives charged over a college admissions scam? They had helped their kids into Ivy League and other elite colleges by paying others to change their kids' test scores or to lie about their sporting prowess. The names included then international law firm co-chair Gordon Caplan, and Desperate Housewives star Felicity Huffman, who was taken in handcuffs at 6am by FBI agents to a downtown Los Angeles detention centre for paying $US15,000 to up her daughter's college entry score.

Gee, thanks Mom. It proved one thing: even glamorous, very well-off people are as terrified as us that they, or their kids, might slip on the ladder. Desperate Housewives star Felicity Huffman has pleaded guilty to charges over the recent college admissions scandal in the US. Credit:AAP There's now a plethora of books aimed at the public unease at what might be going on behind our backs or, indeed, in front of our eyes. As well as Duped and Chamorro-Premuzic's Incompetent Leaders, there is David Robson's The Intelligence Trap about why smart people make stupid – even fatal – decisions (because smart people can be more dogmatic, less humble, he argues). Then there's Winners Take All: The Elite Charade of Changing the World, by Anand Giridharadas, a former McKinsey analyst turned merciless critic. He links the astonishing shifts in wealth over the past few decades to the rise of a certain kind of thinking and behaving that is now pervasive. Giridharadas originally set aside a month to help launch his book; nine months later, and with the book a New York Times best seller, he's still travelling. His argument is that the 0.001 per cent and their associated elites – bankers, management consultants, advisers, financiers, CEOs – who have hoovered up the bulk of the rewards of recent progress have heard, via rebellions, protests and then Brexit and the election of Donald Trump, the sound of tumbrils. Now they claim that they want to do their bit making the world a better place.

There are problems, the major one being, Giridharadas says, that few of them want to change or fix the underlying conditions that have caused most of the problems in the first place – and brought them so much wealth. Instead, he writes, "They favour the use of the private sector and its charitable spoils, the market way of looking at things, and the bypassing of government." Giridharadas names this the Aspen Consensus after the prestigious and wealthy Washington D.C. think tank, the Aspen Institute: "The winners of our age must be challenged to do more good. But never ever tell them to do less harm." Solutions must always be win-win. The author dubs this ecosystem "MarketWorld" and quotes another critic, "It's this idea you can support a health initiative in Nigeria on the Niger Delta, to reduce disease or diarrhoea … and also make an investment in a company that is a polluter in the Niger Delta." (Win-win.) The first televised US presidential debate, between Richard Nixon and John F. Kennedy, revealed “cold, mechanistic figures”, according to one watching elder statesman. Credit:Getty Images How did we get here? There are two turning points, for a start. The September 26, 1960 presidential debate between John F. Kennedy and Richard Nixon was the first time such a debate had been televised. It's regarded as the dawn of the age of television, setting the photogenic, media-savvy Kennedy on his path to the White House. One ageing statesman saw something else: spin, calculation and emptiness. Dean Acheson, former US Secretary of State under President Harry Truman, and one of the architects of the Marshall Plan that sent economic aid to post-war Europe, was appalled. Historian David Halberstam wrote in his book The Fifties: "To him, both candidates appeared to be cold, mechanistic figures who had, with the aid of pollsters and advertising executives, figured out, down to the last decimal point, what stand to take on every issue.

"Do you get a funny sort of sense that, so far at least, there are no human candidates in this campaign?" Dean Acheson wrote to Truman. "They seem improbable, skilful technicians. Both are surrounded by clever people who dash off smart memoranda, but it is not all pulled together on either side, by or into a man. The ideas are too contrived … These two … bore the hell out of me.'" Loading The second turning point, again American. In July 2000, a young man, Gene Kan, sat in front of a US senate judiciary committee investigating "intellectual property in the digital age". Kan's testimony is unsettling, in hindsight, as he articulated a new way of thinking. A 23-year-old software developer, Kan appeared for the digital file-sharing start-ups. Napster had been the first. He acknowledged making some kind of payment to artists but his main concern is the users. "Twenty million Napster users can't be wrong," he told his audience. "Technology moves forward and leaves the stragglers behind. The adapters always win and the stalwarts always lose. Mechanised farming is a good example. You don't see anyone out there with a horse and plough these days." Actually, 20 million people can be wrong, as history agonisingly shows. Nor did moving to mechanised farming involve breaching someone's copyright, that is, effectively pinching someone's work. A later US court case ruled against Napster and shut it down in 2001. Kan committed suicide the following year, but his style of thinking didn't go away. Facebook and Google would not have grown to their size, and have the effect they have had on legacy industries like film, media, publishing and music, if it hadn't been for that attitude. As billionaire investor and philanthropist George Soros told the Davos World Economic Forum in early 2018, the "exceptional profitability" of these digital companies "is largely a function of their avoiding responsibility for – and avoiding paying for – the content on their platforms". He also attacked them, as others have, as monopolies that have become "obstacles to innovation" – the very opposite of what these companies have always told us they stand for.

Anand Giridharadas, a fierce critic of “win-win” culture. Credit:Getty Images Are we complicit or powerless? Obviously, if you're a farmer dudded by your bank or your insurer lied to you, you aren't complicit. If you suffer because your hospital is operating according to a consultancy's notions of efficiency rather than best practice, no. (Back in 2005, businessman and then Patrick Corporation boss Chris Corrigan compared consultants to Mad Hatters who "can construct any result you want and are not burdened with any knowledge". Asked about his views now, Corrigan says he still holds them, adding, "In most cases you do not get what you paid for but certainly what you deserve.") But many of us do know when things don't add up, and we go along with it anyhow. As one resigned observer of the "Canberra consultancy complex" puts it: "People listen to what makes them feel good, not what makes things seem difficult." Still, Giridharadas wrote his controversial book because, after being invited to be a fellow at the Aspen Institute in 2011, he began to feel queasy. There were discussions about democracy and the good society but he flew in private jets and merged with the super-rich in "antler-decorated mansions". He writes, "I began to feel like a … timid accomplice to … a giant sweet-lipped lie." The reception to his exposé was unexpected. "I felt somewhat afraid before publication, thinking there might be a tsunami of resistance," he admits as we talk late one night, he in Maryland, me in Sydney. When he had given the initial talk at the Aspen Institute that led to the book, one private equity man told him he was "an asshole". But now, says Giridharadas, "Great numbers of wealthy people, corporations and foundations have really grappled with [the book's] questions. I don't know if I would have had the fortitude to embrace the criticisms the way they have."

He exclaims, "Partly this is the great gift Donald Trump has given us … We've had 40 years of thinking that the rules of business owners, CEOs and the very rich are the way to run things. Trump is now flamboyantly discrediting that idea." He finishes with hope, "I haven't experienced this in my lifetime. There's an opening right now; we need to pivot away from win-win." He's not the only interviewee to tell me we're at a decisive moment. It's also true that if we don't get this right, we may just end up stuck in slippery hell. Again, we can't say we weren't warned. Albert Brooks, Holly Hunter and William Hurt in 1987’s Broadcast News, which redefined “the Devil” as an attractive, insidious promoter of “flash over substance”. Credit:Alamy Writer and director James L. Brooks has a string of Hollywood credits to his name, including The Simpsons, Taxi, As Good As It Gets and Terms of Endearment. But asked in 2016 what he was proudest of, he replied, "The 'Devil speech' in Broadcast News." That prescient 1987 comedy-drama is about the dumbing-down of television journalism and the speech is delivered by a seriously good reporter who is about to lose his job. He is trying to persuade the woman he secretly loves to beware of another journalist Tom, who is glib, handsome and soaring on the new ethos.