One of the joys of the novel is its endless capacity for reinvention, and 2017 saw fiction writers trying out fresh approaches and new forms. The Man Booker winner was a debut novel from an author with 20 years of short stories under his belt: George Saunders’s magisterial Lincoln in the Bardo (Bloomsbury), in which the death and afterlife of Abraham Lincoln’s young son is told through snippets of civil war memoir and a cacophony of squabbling ghosts, was a fantastically inventive exploration of loss, mourning and the power of empathy. There was an injection of the fantastic, too, in Mohsin Hamid’s Exit West (Hamish Hamilton), which added the device of magical portals opening up across the globe to its spare, devastating portrait of victims of war, creating a singular parable about modernity, migration and the individual’s place in the world.

Jon McGregor has always written about communities; in the acclaimed Reservoir 13 (4th Estate), he deepened his pursuit of a collective voice, encompassing the natural world as well as the human in a cyclical tale of the years going by in an ordinary English village wounded by a girl’s disappearance. Each of Nicola Barker’s books is a world unto itself; with H(a)ppy (William Heinemann), winner of the Goldsmiths prize, she pushed the novel towards objet d’art, using colour and madcap typography to conjure a visionary dystopia of surveillance and control in which creativity and individuality refuse to be constrained.

There were notable returns and some new directions from fiction’s biggest names: Colm Tóibín replayed Greek myth in House of Names (Viking); John Banville channelled Henry James in the Portrait of a Lady sequel Mrs Osmond (Viking); Salman Rushdie went back to realism in The Golden House (Jonathan Cape); and Alan Hollinghurst layered historical snapshots of gay life in The Sparsholt Affair (Picador), a beautifully written chronicle of art and love in a changing Britain.

Jennifer Egan followed up her zippy Pulitzer winner A Visit from the Goon Squad with a more conventional novel of American dreams, Manhattan Beach (Corsair); while Arundhati Roy’s second novel appeared a mere two decades after her first: The Ministry of Utmost Happiness (Hamish Hamilton) was a sprawling, kaleidoscopic fable about love and resistance in modern India. Roddy Doyle excelled himself with Smile (Jonathan Cape), a typically bittersweet novella about a middle-aged man’s memories of his schooldays which pulls the rug shockingly from under the reader’s feet. And in June we said goodbye to the prodigiously talented Helen Dunmore, who died shortly after the publication of her haunting last novel, Birdcage Walk (Windmill), set in an 18th-century Bristol where revolution is in the air.

George Saunders … magisterial. Photograph: David Levene/The Guardian

Of the many classical reboots, the most interesting was Kamila Shamsie’s Home Fire (Bloomsbury Circus), which contrasts the role of the modern state with timeless bonds of love and loyalty by replaying the Antigone myth through the story of two sisters and their jihadi brother. Hogarth Press’s project to novelise Shakespeare continued, with master stylist Edward St Aubyn recasting King Lear as the downfall of a media mogul in Dunbar. Debut novelist Preti Taneja set her fierce, freewheeling version, We That Are Young (Galley Beggar), in contemporary India, with fascinating results.

The trend for autofiction continued, with two scorching novels powered by personal history: Meena Kandasamy’s When I Hit You: or, A Portrait of the Writer as a Young Wife (Atlantic) brutally exposed the violence of an abusive marriage within the constraints of Indian society; and The End of Eddy by Édouard Louis, translated by Michael Lucey (Harvill Secker), is a savage account of growing up poor, gay and victimised in rural France. Meanwhile, the intimate horrors of a toxic marriage – and toxic parents – were skewered in Gwendoline Riley’s pin-sharp First Love (Granta).

Two slim, terrifying volumes lingered in the mind: Fever Dream by Argentinian writer Samanta Schweblin, translated by Megan McDowell (Oneworld), was a gloriously creepy fable taking in bodyswapping, maternal dread and the dangers of GM crops. In Such Small Hands by Andrés Barba, translated from the Spanish by Lisa Dillman (Portobello), a traumatised young girl’s arrival in an orphanage is the trigger for an explosion of love, hate and repressed desire. Both are quickly read, never forgotten. Another gem in translation was Olga Tokarczuk’s Flights, translated by Jennifer Croft (Fitzcarraldo), which in the vein of WG Sebald knits together snippets of fiction, narrative and reflection to meditate on human anatomy and the meaning of travel: this is a delicate, ingenious book that is constantly making new connections.

The Lie of the Land.

If you’re after the juicier pleasures of a realist pageturner, pick up Amanda Craig’s canny portrait of the bitter divisions in a marriage and the UK: The Lie of the Land (Little, Brown) sees a privileged couple who feel too poor for London move to rural Dorset. There they discover how the other 90% live in an elegantly written expose of all the things the elite would rather not consider about poverty, inequality, food production and class, with a nailbiting mystery thrown in.

Debuts to celebrate included Yaa Gyasi’s ambitious, multi-generational saga of the effects of the slave trade, Homegoing (Penguin); Gabriel Tallent’s intense tale of abuse and self-determination in backwoods California, My Absolute Darling (4th Estate); and Sally Rooney’s witty anatomisation of modern attitudes in post-crash Dublin, Conversations With Friends (Faber). Former US soldier Brian Van Reet’s Spoils (Jonathan Cape) was a brilliantly written account of kidnap and conquest in the early stages of the Iraq war; for the Iraqi perspective, turn to Muhsin al-Ramli’s The President’s Gardens, translated by Luke Leafgren (Maclehose), which follows a group of friends growing up under Saddam Hussein.

The year’s short stories tended to the dark and the disturbing: standout collections included Sarah Hall’s Madame Zero (Faber), elegant tales of sex, motherhood and transformation, and June Caldwell’s Room Little Darker (New Island), a supercharged gothic debut from an Irish writer to watch. Another debut, Eley Williams’s Attrib. And Other Stories (Influx), cornered the market in cerebral playfulness.

Madame Zero.

Finally, two novels that make fitting reading for the season. Bernard MacLaverty’s Midwinter Break (Jonathan Cape), about an elderly married couple on a mini-break in Amsterdam, may put you off the festive sherry: its portrayal of the tightening vice of alcohol addiction is unparalleled. But its profound exploration of love, companionship, faith, work and our search for meaning in life made it a tender masterpiece, and one of the year’s essential reads.

Ali Smith’s Winter (Hamish Hamilton) is the second in her seasonal quartet, following the Booker-shortlisted Autumn, and once again tackles the biggest subjects with the lightest touch. A capacious, generous shapeshifter of a novel taking in Greenham Common and Barbara Hepworth, Shakespeare and global migration, it juxtaposes art with nature and protest with apathy, finding surprising alliances in a family riven by feuds. It’s a book with Christmas at its heart, in all its familiarity and estrangement: about time, and out of time, like the festival itself.

• What have you enjoyed reading in 2017? Send your choices in 150 words or fewer to readers.books@theguardian.com or Readers’ Books of the Year, Review, The Guardian, Kings Place, 90 York Way, London N1 9GU (including address and phone number), to arrive no later than Monday 11 December. Or comment below. We will publish a selection in the paper and online on 30 December.