When I started at the New Statesman in December 2010, we were six months into Britain’s first experience of coalition government since the 1940s, a surprisingly strong alliance between David Cameron’s Conservatives and Nick Clegg’s Liberal Democrats. Since then, we’ve had two more general elections, a Scottish independence referendum and a referendum on our membership of the EU. In both the general elections I’ve covered, as well as the 2015 Labour leadership contest and the US presidential election, the working assumption of most political journalists about the result has been wrong. That is not only embarrassing; it has grave implications for our trade.

Historians warn about “the teleological view of history” – assuming a fixed endpoint and then telling the story as if it was always heading for that end point. Something similar has happened, over and over again, in political journalism.

In 2015, we assumed that the Conservatives could not win an overall majority. The likeliest result, most journalists thought, was a Labour minority adminstration backed up by the SNP. That affected the questions we asked: Ed Miliband, would you do a deal with the Scottish Nationalists? Would you give the SNP another referendum? How much money would you give to Scotland to get their votes? And it also affected the Conservatives’ “squeeze” message to bolster their own support.

One advert showed Ed Miliband in Alex Salmond’s pocket. In the English seats the Tories needed to win, it was an extremely effective message. What happened feels like a version of the Hawthorne effect in science, where study participants alter their behaviour because they know they are being observed. By focusing so strongly on the possibility of a deal with the SNP, journalists made it look extremely likely. That, in turn, drove voter behaviour.

It wasn’t only journalists who assumed the wrong endpoint. In the years since, senior Tories have admitted that the Conservative manifesto was written with the idea that some of its policies would have to be junked in a horse-trade with the Lib Dems.

Cut to election night 2015. The exit poll arrives – and it’s an overall majority, albeit it a slim one, for David Cameron. All those questions about the SNP were not just redundant, but might have influenced the outcome. At the same time, insufficient consideration was given to what the Conservatives might do with a majority.

The answer was an EU referendum, long demanded by a small but monomaniacal subset of the Conservative parliamentary party. Covering the build-up to that, once again political journalism struggled because it assumed the result – Remain – and worked backwards from there. In the course of the campaign, I can remember hardly any consideration being given to what form Brexit would take: there was no appetite for discussions of the merits of “Norway plus” against a Canada-style trade deal. Leaving the EU was deemed unlikely to happen, and therefore not sufficiently interrogated. That has had enormous repercussions ever since.

“Brexit means Brexit” said Theresa May. But during the referendum campaign, the Vote Leave campaign spokesmen (and they were mostly men) were extremely hazy on the form that leaving might take.

It took until 8 May 2016 for the BBC’s Andrew Marr to get a definitive statement on Vote Leave’s position on our future membership of the single market from Michael Gove:

AM: Let me ask you, just before we leave the economics actually, a very simple question I have tried to get an answer to from various people on your side – is should we or should we not be inside the single market? Do you want us to stay inside the single market? Yes or no.

MG: No. We should be outside the single market. We should have access to the single market, but we should not be governed by the rules that the European Court of Justice imposes on us, which cost business and

restrict freedom.

If the single market received little attention, what about leaving the customs union? The website FullFact states: “Leave campaigners hardly mentioned the customs union in explicit terms at all, so there was generally little clarity about what leaving might mean in that regard.”

As for the fabled “WTO terms” or leaving without any deal at all, these were barely mentioned. It would be extremely useful now to have on-the-record statements from 2016 from the Tory leadership candidates on this issue. We could then see how far their positions have moved in just three years, and how much more extreme the Conservative position has become.

Let’s go across the Atlantic. There aren’t many things that Donald Trump and Jeremy Corbyn have in common, but here is one. Both were treated as joke candidates at the start of their campaigns. And joke candidates don’t face the same level of scrutiny as front-runners – which is a problem when they turn out to be serious contenders.

When I wrote about post-truth politics for Nieman Reports in 2016, the New Yorker journalist Evan Osnos told me: “For many months there were reporters who were still too light-hearted about the Trump phenomenon, long after it should have been plain to them that it was not remotely funny. It was a mistake to allow him to go on television, month after month, phoning into interviews that would ordinarily require the person to be in the studio and subject to the kind of scrutiny that an in-person interview produces. But instead, because he was treated as something between a joke and a boon for ratings, he was allowed to call in. That was an abdication of responsibility.” Even once it was a straight fight between Trump and Hillary Clinton, too often she was treated like the next president and he was treated like the entertainment. There was a double standard of scrutiny.

A similar pattern was true of Jeremy Corbyn in 2015. He started as a 100-1 outsider, and it was only when constituency Labour parties began to return their endorsements that early polling figures putting him way ahead were vindicated.

From then on, the coverage focused on the unlikelihood of his candidacy – the idea that a 60-something lifelong rebel backbencher could become leader of the opposition. It was a kind of fairy tale. But that strong personal story squeezed out consideration of his positions: anti-America and sympathetic to Russia; opposed to Nato; sceptical of the EU; willing to talk to Hamas and the IRA but less so to, say, Israeli politicians.

****

I wonder whether I got it wrong during this period. I was not a fan of Corbyn from the start, precisely because I knew the strain of left-wing politics he represented, and it was not the same as mine. But because I opposed him openly – in admittedly caustic terms – did it cost me any credibility to criticise him? Was there a way to have covered his early candidacy that was both fair to him and his appeal, and took into account the views that would later make him so polarising?

It’s a deeply personal question, because I have friends who were early Corbyn supporters and now find his position on the EU – his willingness to enact Brexit, as long as it’s a “Labour Brexit” – incomprehensible. Why didn’t anyone warn us he was a Eurosceptic, they complain. We did and you wouldn’t listen, I silently reply.

Again, the problem comes at least partially from the teleological view of politics. With the EU referendum months and months away – and a win for Remain expected – Corbyn’s Euroscepticism seemed like a minor quirk, rather than a defining part of his politics. Journalists failed again to make the case for what really mattered.

In 2017, I was determined not to make the same mistake again. Talking to Stephen Bush – now the New Statesman’s political editor – about our coverage of the election campaign, we vowed to keep our minds as open as possible. Looking at the headline polls, it seemed as though Theresa May would storm to victory, increasing her majority from 12 – to what? 50? 100? 150?

We decided that whatever the polls said, we would interrogate the other, more unlikely possibilities. Our newest writer, Patrick Maguire, was commissioned to write a series of online pieces on the Democratic Unionist Party (DUP) of Northern Ireland. Why had senior DUP members come to the Conservative Party conference the previous year? They were clearly being lined up as potential partners – not for a coalition like the Lib Dems, but a looser “confidence and supply” situation. That meant they would support the government on Budgets and motions of no confidence. Everything else was negotiable.

During the campaign itself, Stephen took a hard look at the polls and wrote a story about how Corbyn’s numbers were improving over the election period. On 11 May, he published a piece headlined “Unnoticed and unreported, Jeremy Corbyn is surging in the polls”. It did extremely well on Facebook – thanks to a boost from the left-wing campaign group Momentum – but made little impact in “the Westminster village”.

There were other signs that May was in trouble. A week before the election, the YouGov seat predictor got the result – a hung parliament – pretty much dead-on. It led to a front page write-up by Sam Coates of the Times. I remember messaging him: “Sam, either that projection is bonkers, or I am. I no longer know.” I take some consolation from the fact that I was remaining open-minded. In the war of facts against narrative, the narrative continued to be that the Conservatives would increase their majority. It was tempting to discard any facts that did not fit.

A few days before the election, I had possibly my greatest ever commissioning triumph. We had jokingly talked in the office about “hope clicks” – the opposite of “hate clicks”. We didn’t want to produce journalism that simply validated our readers’ preconceptions – but equally, it was notable how much interest there was in stories which deviated from the standard narrative of Theresa May marching to victory. Tell me, I said, what will happen in a hung parliament? Our readers should know. Let’s write it. The worst case is that a few of them will learn a bit more about coalitions, confidence and supply, and minority administrations.

I found out the best case at 10.01pm on 8 June 2017. Minutes before the exit poll, people were still excitedly tweeting about the possibility of a 100-seat majority for Theresa May. Then David Dimbleby delivered the news on BBC One. The Conservatives had lost seats: they remained the largest party, but no longer had an overall majority. There’s a photo of my face reacting to the exit poll: I am shocked. We all were. And pretty much that very second, “What happens in a hung parliament?” became one of the most searched-for terms in the UK. Hundreds of thousands of people were reading our piece at once. I was happy – it was a great article. I was also struck by something. As a specialist website, the New Statesman often struggles to compete with the huge newspaper websites on Google search results. Here, we had triumphed… because, it seemed, our rivals hadn’t written the piece.

The next day, Theresa May unveiled the price of staying in Downing Street – a confidence and supply arrangement with the DUP. We later discovered it would cost £1bn in extra cash for Northern Ireland. £1bn of public money, something which never figured in the election campaign, unlike the SNP squeeze two years before.

I won’t be covering another election for the New Statesman as I am about to leave the magazine, but I hope that my colleagues will continue their proud tradition of ignoring the herd – and therefore increasing their chances of being right.

The seductive power of the conventional narrative is one of the most distorting forces in political journalism. Jeremy Corbyn is useless, Donald Trump is a joke, Theresa May is the Iron Lady, Remain will win, the Liberal Democrats are finished, Nigel Farage has retired from politics. All of these seem true, until – suddenly – they are not. For commentators and reporters on the left, that is particularly tricky terrain to navigate, because the printed press is dominated by the right, and therefore the consensus tends to be sympathetic to that point of view.

For me, indulging in the teleological view of history is the first deadly sin of political journalism. It is also the easiest to cure – just stop doing it! Ask questions even if they seem odd or niche. Pin down politicians on under-considered scenarios. We must try to tune out what everyone else is obsessed with, and ask ourselves: what could happen that no one is talking about?



Let ‘em hang: Helen Lewis and Ian Hislop at the New Statesman’s election night party in 2017

****

As there are traditionally seven deadly sins, here are six others. To draw them up, I talked to some of the political journalists whom I most respect, and who are trying to improve the quality of their trade.

Innumeracy

Most political journalists are arts and humanities graduates. That can lead to a lack of ease with economics as a policy area, with reports that involve figures, with real-terms and with parliamentary arithmetic – how many votes are needed for this to pass? It also leads to a lack of understanding in the way polls are reported. Most have a margin of error of 3 percentage points. All are an art as well as a science, taking the raw numbers and applying filters (such as likelihood to vote). Still, though, outlying polls are reported as news without being placed in context, and moves of 1 or 2 percentage points are treated as significant.

The “neutral amplifier” model

One of my favourite sayings about journalism is this: “It’s not journalism’s job to report that people are saying it’s raining. It’s journalism’s job to look out of the window.” Too often, the words of a high-profile politician are repeated uncritically: “I will renegotiate the withdrawal agreement and get something better”, “we will leave the single market but retain the same benefits”.

In other cases, a clearly partisan source is treated as (anonymous) gospel because the story is deemed too good to debunk – and in any case, the person briefing might then go to a rival outlet with it. Some journalists are seen by MPs and special advisers as reliable relays for spin who won’t ask awkward questions. “We’ll use [X] when we want projection,” is how that dynamic is phrased.

But the job of journalists is not to tell you what the Labour spokesman and “sources close to Michael Gove” are saying. It’s to evaluate their words and deliver them to readers or viewers in context. We need to look out of the window.

A related problem is the rise of journalists as personal brands. Twitter has created an arena where status – measured in follower numbers – is obvious even to outsiders. That has gamified political journalism. Reporters want to get full credit for their own stories, attract attention to themselves personally and (sometimes) buff their egos. Tweet early, tweet often and don’t worry too much if you get something wrong, the thinking goes, because any attention is good.

The Confident Posh Man problem

This phrase was coined by a lobby journalist I know who self-defines as a Confident Posh Man. There are loads of them in Westminster, he observes. There are two obvious reasons: the first is that lobby journalists, being based in the Commons, double up as fixers and lobbyists for their papers with senior politicians. If a media organisation wants a cabinet minister to speak at a conference, it’s often the lobby correspondent’s job to put in the request to the relevant special adviser. It would be a brave, or confident, news desk that sent someone with blue hair and visible tattoos.

The Commons is extremely formal, insisting on men wearing ties in the chamber, and is an intimidating place to work. Stella Creasy, the Labour MP, calls it “Hogwarts”. The strange quirks – prayer cards, Pugin carpets, tearooms serving spotted dick – are less disorienting if you already attended a public school.

The second reason is that news desks each send one to six correspondents, so there is no way to co-ordinate gender balance, let alone racial diversity, across the entire lobby. As a result, the lobby is very white and dominated by graduates and the privately educated. It has plenty of younger women in it, but the attrition rate is high after 30, because late nights, long days and unpredictable hours are hard to juggle with caring responsibilities. It was only in 2015 that the Guardian announced the first ever job share for political editors (although that has already ended).

No punishment for failure

Political journalism is often speculative, so there has to be latitude for predictions that prove incorrect. Sometimes that is taken as licence to write stories that are unfalsifiable, or will be correct – eventually. “Theresa May on the brink as Tory MPs revolt” was true for nearly two years after the 2017 election before she finally resigned. One Sunday paper reported that 48 letters expressing no-confidence in her leadership had gone into Graham Brady, chairman of the 1922 committee, a year before they did so. (“An amazing scoop, if you look at it one way,” said a writer for another organisation, dryly.)

Among the journalists I spoke to, there was a general acceptance that some of their colleagues treated political journalism as entertainment. A story could be just that. At outlets such as the Financial Times and Bloomberg, which cater to a specialist audience who may make investment decisions based on their coverage, there are heavy sanctions for getting it wrong. Elsewhere, attitudes are more relaxed.

99-1 as balance

Many of the problems I’ve described are more acute on newspapers and websites, because of the time pressures and focus on headlines. This one is worse in broadcasters, which have a duty under regulatory rules to be impartial. After commissioning an expert report, the BBC has ruled that climate change is a scientific fact, and so the starting point for discussions is no longer “is it real or not?” A briefing note sent to staff in September 2018 told staff: “You do not need a ‘denier’ to balance the debate.”

Yet during the EU referendum campaign, I heard frequent complaints that the economic risks of leaving the EU – the subject of broad agreement by 99 per cent of economists, trade experts and scientists – were treated as impossible to rule on. One economist, usually Patrick Minford of Cardiff University, would represent the small “it’s fine” camp. One economist would represent the entire rest of the profession. But viewers and listeners would have no idea that one spoke for a much larger group than the other.

The view from Versailles

As humans, we find people more interesting than policies. But that has huge distorting effects. The best political journalists use people to tell their stories, reducing an abstract clash of ideas to a human scale. The worst ones treat existential questions as props for a Punch and Judy show. It was this tendency which led the EU referendum campaign to be covered as a contest between “Dave and Boris”.

What can we do about this, I asked one journalist I respect. His answer surprised me. “People will read about clashes,” he said. “They won’t read about the dynamics between two theories.” Political journalism was “not like a murder. It’s like two sides of a debate, but you’ve decided what the debate is, and what the sides are. If you don’t have the confidence to do that, you have nothing.”

His answer, he said, was to “create people” – that is, to build up politicians as emblems of a particular viewpoint. Middle-of-the-road MPs have traditionally been reluctant to talk to journalists. The whips don’t like it; such MPs worry it will harm their careers. The “awkward squad” – the fringes of the party – behave differently. “Jacob Rees-Mogg exists because he picks up the phone to journalists,” he told me.

But what that means is political shows where Anna Soubry debates John Redwood, forever. My unnamed source encouraged a moderate Tory MP to do more broadcast interviews, and to present himself as the spokesman for the broad centre of his party. It was an act of chutzpah, but perhaps a necessary one. “All our stories are told through other people’s mouths,” my source said. “Is creating someone to deliver those quotes overstepping the mark?”

He thought not. And the more I’ve thought about it, the more I agree with him. Journalists “create” public figures all the time. We decide which YouTuber to interview, which athlete to put on the front page, and which politicians are at the top of our mental Rolodex. We are, quite rightly, beginning to reckon with the lack of race, class and gender diversity in political journalism, and to expand our pool of sources. Why shouldn’t that apply on ideological grounds too?

There is another dark side to the view from Versailles, and it is this. Too often, I think, journalists identify one type of person with “true Britons”. We talk about Labour’s “heartlands” in working-class northern English regions, when the party is now strongest in multi-ethnic big cities. Asked to reflect on Labour’s loss of votes in the European elections, Richard Tice, chair of the Brexit Party, said: “This has happened because they haven’t listened to their core heartlands, they’ve listened to people in Islington.” But Labour has two Westminster seats in the Islington region, each with a thumping majority. So where is the party’s heartland now?

Because political journalists are stuck in the Commons and on Twitter – real world and digital versions of Versailles – they are prone to panic about not representing “real people”. But who gets depicted as the authentic voice of unheard Britain is governed by implicit assumptions that are grim when exposed to the light. Why are the views of a retired steelworker in Grimsby about “where the country has gone wrong” more important than those of a second-generation Nigerian-British nurse in Plaistow? Citizenship is supposed to transcend personal identity, and yet we still indulge an idea of the “volk”. This tendency applies equally in America, where retired steelworkers in Pennsylvania were held up as the Great Unheard, but the same epithet was not bestowed on black voters in Detroit.

The scattered successes of the far right across Europe are deemed to tell us something about what “real people” are thinking, in the way that the more quiet triumphs of Green politicians are not. Across Europe, the story of the latest EU elections was the rise of environmentalist parties. But somehow, their voters aren’t held up as “real people” whose real concerns must be heeded by the main parties. Why not?

****

Political journalism is incredibly difficult. It involves cultivating a source, sometimes over years, with the knowledge that a single rogue story could result in them freezing you out altogether. Party leaderships can be uncooperative, refusing to submit to print interviews or press conferences. In the age of Twitter and rolling news, speed is highly prized. Political journalists work long, antisocial hours, in a building full of mice. They get trolled on Twitter. At least one, the BBC’s political editor Laura Kuenssberg, has needed bodyguards just to enable her to do her job.

On print titles, they answer to news desks that want a story that is new, exclusive and can be summed up in a single sentence. Trying to mash the complexity of reality into that format is incredibly tough.

On television and radio, they struggle to navigate their way through rules on impartiality in a hyper-partisan environment where both sides constantly complain about bias. They are expected to understand a dizzying array of jargon, and to have the chutzpah to tell a cabinet minister, live and in real time, that they are talking bollocks. That takes serious confidence and expertise. They deal with liars, self-promoters and sources who are actively trying to manipulate them.

There is excellent political journalism out there. And god, do we need it now more than ever.

This piece is adapted from a lecture delivered at the Reuters Institute for Journalism, Oxford University