When Anna Burns, author of last year’s Booker prize-winning account of the Troubles, Milkman, was asked whether writing was a political act, she was taken aback. “Honestly? This is the sort of question I don’t know what to do with. It’s not how my brain works.” Eventually she allowed that if politics was about power then yes, OK, her work was political. Such qualms did not deter the judges of the inaugural Orwell prize for political fiction from awarding Burns another trophy. Chair of judges Tom Sutcliffe praised Milkman’s “account of how political allegiances crush and deform our instinctive human loyalties”.

Like the rest of the Orwell prize shortlist, Milkman has a theme rather than an agenda. Always capacious, the genre of political fiction can now accommodate authors such as Ali Smith, Rachel Kushner, Paul Beatty and Jonathan Coe. As George Orwell wrote: “No book is genuinely free from political bias. The opinion that art should have nothing to do with politics is itself a political attitude.” Much harder to find, however, is an example of what one might call the campaigning novel: that subset that includes classics by the likes of Charles Dickens and Émile Zola alongside fiction-cloaked manifestos, memoirs and works of reportage. What unites them is a passionate desire to use character and narrative to draw the reader’s attention to some social ill and to galvanise efforts to remedy it. As Sam Leith, Orwell prize judge, describes the approach: “Look at this, isn’t it awful?”

When he announced the longlist in May, Sutcliffe said that it “acknowledges that the politics in a book can often be found between the lines, rather than on them”. Coe, whose state-of-the-nation novel Middle England did not make the longlist, took this as an “implicit rebuke” to books with an explicit agenda. “My response would be that sometimes there is a time to speak loud and clear,” he says. “And anyway, as my character Doug Anderton says in The Rotters’ Club: ‘Subtlety is the English disease.’ I think it’s a peculiarly English thing, this recoiling from a novel whose political message seems too overt. In France, for example, writers can receive extra points for being engagés. Even in Scotland, the likes of James Kelman and Alasdair Gray are considered national heroes partly for the political content of their work. But in England that seems to be considered rather vulgar.”

I think it’s a peculiarly English thing, this recoiling from a novel whose political message seems too overt Jonathan Coe

Manifestly political novels have always aroused some degree of suspicion. Orwell famously categorised Harriet Beecher Stowe’s anti-slavery novel Uncle Tom’s Cabin (1852) as a “good bad book” – crude yet effective – and Milan Kundera in turn dismissed Orwell’s Nineteen Eighty-Four (1949) as “political thought disguised as a novel”. In a devastating review, Whittaker Chambers said that Ayn Rand’s colossal philosophical tract Atlas Shrugged (1957) “can be called a novel only by devaluing the term … Its story merely serves Miss Rand to get the customers inside the tent, and as a soapbox for delivering her Message.”

Resistance to a message has only grown stronger since then. Leith believes the tides of literary fashion have left the campaigning novel high and dry. “We’ve all drunk in the idea that you don’t reward a book for its subject matter but its literary craft,” he says. “While the consistency of the metaphors is fair game, the value of their ideas is less so.” This critical focus on form and language rewards a book as unusual as Milkman but creates a frosty climate for any author with a blatant reformist agenda.

“When you study literature, you learn the mantra that great art is not political,” agrees Joanna Kavenna, whose latest novel Zed is a wry satire about the unravelling of a supranational tech giant. “You read that ideas are somehow an imposition on fiction. But as soon as you create a character or write a story, you’re into an idea about society whether you like it or not.” Influenced by Kafka, Gogol and the “tradition of unease”, Kavenna prefers to deploy irony to express the strangeness of modern life. More didactic fiction, she says, is prone to being dismissed as propaganda. “Writers like [William] Morris could be very certain: this is my political programme, I want these outcomes. They were so sure of themselves and where they were in society. Our world is much more volatile and the audiences are much more sophisticated. People are so wary of blunt polemic – quite rightly – that we question the motives of everything all the time.”

Facebook Twitter Pinterest Anna Burns’s Milkman is set during the Troubles in Belfast, Northern Ireland, in the 1970s. Photograph: Alain Le Garsmeur/Getty Images

It wasn’t always the case that readers recoiled from proselytising fiction. Take, for example, Looking Backward: 2000-1887 (1888). Wrapping his prospectus for American socialism inside the tale of a Bostonian aristocrat who sleeps for 113 years and awakes in an egalitarian utopia, the journalist Edward Bellamy produced a colossal bestseller that was said to have introduced more Americans to socialism than Karl Marx. Looking Backward’s influence extended to suffragists, populists, HG Wells, Morris, Franklin D Roosevelt and Clement Attlee, who told Bellamy’s son Paul that the postwar Labour government was “a child of the Bellamy ideal”. Bellamy did not transform the world to the extent that he had hoped but he certainly left it somewhat different.

Campaigning novels aim for the statute books and occasionally hit their mark. The Jungle (1906), Upton Sinclair’s stomach-churning exposé of the plight of immigrant labourers in the Chicago meatpacking industry, prompted President Theodore Roosevelt to establish the forerunner of the Food and Drug Administration, though not (to Sinclair’s disappointment) to improve workers’ rights. The Citadel (1937), AJ Cronin’s bestseller about a morally compromised doctor in a Welsh mining town, influenced his friend Aneurin Bevan’s plans for the NHS. Beecher Stowe’s Uncle Tom’s Cabin fired up the abolitionist movement. The Ragged-Trousered Philanthropists (1914), Robert Tressell’s autobiographical account of poverty and exploitation in small-town England, has, as Labour MP Dan Carden said recently, “given the socialist movement of our time a story to get behind”.

At the other end of the political spectrum, Rand’s hectoring libertarian parables The Fountainhead (1943) and Atlas Shrugged have shaped the thinking of powerful disciples including the former House speaker Paul Ryan, US supreme court justice Clarence Thomas, the economist Alan Greenspan and chancellor of the exchequer Sajid Javid. Chinua Achebe’s anti-colonialist milestone Things Fall Apart (1958), a favourite of Jeremy Corbyn and Barack Obama, and Aleksandr Solzhenitsyn’s gulag chronicle One Day in the Life of Ivan Denisovich (1962) alerted an international readership to gross injustices in Nigeria and the Soviet Union respectively. These are just some examples that made a significant dent in the political landscape; many more novelists thought it worth trying. But the campaigning novel has become an anachronism. A certain kind of urgent, unsubtle voice has fallen quiet.

Facebook Twitter Pinterest ‘No book is genuinely free from political bias’ … George Orwell. Photograph: BBC

Jean Seaton, director of the Orwell prize, suggests there is a political dimension to the campaigning novel’s postwar fall from grace. Many of the most politically effective prewar novelists were socialists, whose moral prestige was destabilised by Stalinism. Stalin’s insistence on socialist realism made polemic both aesthetically and morally suspect. “Bluntness was done for by communism because communism told you that you had to write novels that had messages,” Seaton says. “And we were all on the side of Vasily Grossman, who didn’t write messages. Bluntness looks ideological.”

Any novelist who does still want to stimulate political reform, the critics be damned, has to reckon with a political class that takes far less interest in literary fiction than the generation of Attlee and Bevan. Seaton recalls spending time with Labour cabinet minister Roy Jenkins during the 1960s: “He took ideas in and metabolised them. There is something about the lives that senior politicians and administrators have now that doesn’t reward hinterland and doesn’t give them time to reflect.” A vicious circle takes effect, in which politicians lose interest in political fiction and authors lose faith in their potential to influence politics.

Uncle Tom’s Cabin, Looking Backward, The Grapes of Wrath: all runaway bestsellers that demanded a response from politicians

One notable exception is Labour party deputy leader Tom Watson, whose touchstones include Looking Backward and Arthur Koestler’s eye-opening account of Stalinist persecution, Darkness at Noon (1940). “The best politicians capture the imagination, as do all great novels,” Watson says. “They can touch the emotional self as well as the rational self. They can stimulate internal thinking. Darkness at Noon gave me a real insight into how ideology can be used to oppress.”

The average politician wouldn’t be wrong, however, to assume that political fiction lacks traction with voters and therefore a pressing claim on their attention. No recent book comes close to the reach of Uncle Tom’s Cabin, Looking Backward or John Steinbeck’s dustbowl tragedy The Grapes of Wrath (1939): all runaway bestsellers that demanded some kind of response from politicians. This reflects the novel’s broader loss of cultural primacy. A century ago, it was still the dominant form of storytelling so any writer with a political message to disseminate was inclined to give it a shot. Campaigning writers believed that a veneer of fiction, however thin, was deemed necessary to sugarcoat the message. After the second world war, however, the evolution of the “non-fiction novel” removed the need for journalists to masquerade as novelists. A 21st-century Sinclair might well decide to frame a book like The Jungle as straightforward reportage, while a modern Looking Backward would probably be a manifesto in the style of Paul Mason’s PostCapitalism (2015). The popularity of mass-market non-fiction has relieved the old commercial pressure to crunch a political argument into an artificial format. John Lanchester, for example, alternates between fiction and non-fiction according to what he wants to say.

Facebook Twitter Pinterest ‘A novel for television’ … The Wire. Photograph: Allstar/HBO

Why even stick to the printed page? Its creator David Simon called The Wire “a novel for television” and critics responded by saying that it did for Baltimore what Dickens did for Victorian London. Like Sinclair, Simon made his bones as a campaigning journalist and brought a certain didacticism to The Wire. “We were the editorial board,” he said of the writers’ room. “And we were making our arguments as best we could, utilising narrative drama.” It’s hard to think of a modern novel with the moral zeal and political impact of Ken Loach’s films Cathy Come Home (1966) or I, Daniel Blake (2016): this year’s Orwell prize for best article went to an essay called “Administrative Justice in the Wake of I, Daniel Blake”. Perhaps it is fine if Simon is our Dickens and Loach our Tressell. “There’s no absolute necessity for novels to enjoy a position of privilege in the culture,” says Leith. “In terms of how ideas get disseminated, whatever medium is reaching people is the important one.”

Even now, there are still a few novelists who set out to grab the reader by the lapels. Seaton commends Dave Eggers’ urgent Silicon Valley satire The Circle (2013) for crystallising concerns about social media in the form of a thriller and causing ripples in the worlds of tech and politics. Frank Cottrell-Boyce, the children’s author and screenwriter, says that YA fiction, with its naturally curious and idealistic readership, is an underrated vehicle for political ideas and compares the well-travelled novelist Elizabeth Laird to “a particularly passionate and empathetic foreign correspondent, reporting back from the wider world”. But he’s concerned that, since the success of Suzanne Collins’s The Hunger Games (2008), dark futures have eclipsed bright ones.

Facebook Twitter Pinterest Moral zeal … Ken Loach’s 2016 film I, Daniel Blake.

“I worry that the flood of dystopian fiction has changed the definition of normal – that it has acclimatised younger readers to an atmosphere of corruption and lying,” he says. “I do believe that one important duty which fiction seems not to be fulfilling is painting a picture of what good looks like. I guess to be truly political a work of art should be offering at least the possibility of change and a way forward.”

All literary traditions have peaks and troughs but the campaigning novel seems to have been sunk from all sides, more often than not regarded with apathy or scepticism by writers, readers, critics and politicians alike. Coe feels that politically motivated authors will continue to be forced to find more circuitous routes to the reader’s sympathies, even when they would rather cut to the heart of the matter.

“Two of my most overtly political novels, What a Carve Up! and Number 11, are both quite tricksy in their approach,” he says, “almost as if you have to apologise for the political content by wrapping it up in something irrefutably ‘literary’.” Ken Loach read Number 11, Coe says, and “wrote me a very kind letter in which he remarked that all of the review quotes on the paperback seemed to be praising the literary aspects of the book and made little mention of the fact that it was a novel with a strong anti-austerity message. It was a comment that resonated with me, and made me suspect that in this country the best a novel can hope for is to be praised in spite of its politics, not because of them.”

• Dorian Lynskey’s The Ministry of Truth: A Biography of George Orwell’s 1984 is published by Picador.

• This article was amended on 3 September 2019 to correct an editing error that led to the suggestion that it was Franklin D Roosevelt, rather than Theodore, who established the forerunner of the US Food and Drug Administration.