We thumb through a year that saw a US author win the Booker, the death of a short-story master – and the first Brexit novel • Vote: What was your favourite book of the year?

“It was the worst of times, it was the worst of times ...” From its opening line Ali Smith’s Autumn (Hamish Hamilton), written and published at speed to become the first Brexit novel, faced up to the sometimes despairing mood of Britain in 2016 with humour and grace. This being an Ali Smith novel, it also found solace in the consolations of friendship and art, spinning a typically lightfooted meditation on mortality, mutability and how to keep your head in troubled times around the tale of an uncertain young woman and her elderly childhood friend.

But times were good for fiction: this was a rich 12 months, with plenty of big names and big ideas – though not always wrapped up in the same package. The year began with an elegant portrait of Shostakovich’s life under Stalin from Julian Barnes, The Noise of Time (Jonathan Cape), and an involving cold-war spy story from Helen Dunmore, Exposure (Windmill). Other big hitters included Don DeLillo with Zero K (Picador), a chilly investigation into cryogenics and father-son relationships; JM Coetzee’s drily philosophical inquiry The Schooldays of Jesus (Harvill Secker); and Javier Marías delving into desire and guilt in Thus Bad Begins (Penguin, translated by Margaret Jull Costa). Rose Tremain was on top form with her nuanced analysis of emotional and political neutrality, The Gustav Sonata (Chatto & Windus), while AL Kennedy tenderly anatomised London and loneliness in Serious Sweet (Jonathan Cape) and Ian McEwan had fun in Nutshell (Jonathan Cape), a slightly crusty jeu d’esprit whose foetus narrator shows a precocious appreciation of poetry and fine wine.

Facebook Twitter Pinterest Back with some brilliant writing in Swing Time … Zadie Smith. Photograph: Brian Dowling/Getty Images

Towards the end of 2016 came Sebastian Barry’s outstanding Days Without End (Faber), an intense, visceral novel that brings the violence and terror of 19th-century America to blazing life; and Zadie Smith’s more downbeat Swing Time (Hamish Hamilton), exploring friendship and failure, isolation and identity, distraction and the power of dance – what it’s like to look back at where you came from, and to not know where you want to go. Smith’s fifth novel may not have dug as deep as her previous books, but contained some brilliant writing about growing up in London.

In between, the penultimate volume of Karl Ove Knausgaard’s autobiographical novel cycle, Some Rain Must Fall (Vintage, translated by Don Bartlett), zeroed in on his 20s and the struggle to invent himself as a writer. And novelists continued to reinvent Shakespeare, with Howard Jacobson, Anne Tyler and Margaret Atwood refettling The Merchant of Venice, The Taming of the Shrew and The Tempest respectively; Atwood’s typically playful Hag-Seed (Hogarth), with Prospero a theatre director staging The Tempest in a prison, was the highlight.

There was a wealth of US fiction, ranging from Annie Proulx’s vast ecological saga about the environmental degradation of the last 300 years, Barkskins (Fourth Estate), to Elizabeth Strout’s slim consideration of mother-love, My Name is Lucy Barton (Viking). We had new books from Edmund White and Jay McInerney, Dave Eggers and Lionel Shriver; while Viet Thanh Nguyen brilliantly explored the legacy of the Vietnam war in the Pulitzer-winning The Sympathizer (Corsair), and Jonathan Safran Foer published his first novel in a decade. The sprawling Here I Am (Hamish Hamilton) ranges over such weighty subjects as America, Israel, marriage and masculinity, with diversions into obscene uses for a doorknob.

And in the third year of eligibility, a US author won the Man Booker prize for the first time. Paul Beatty’s The Sellout (Oneworld), a no-holds-barred satire about the history and legacy of racism in America, had clocked up 18 rejections from cautious publishers. This cerebral rollercoaster ride of a novel was praised to the skies in the US but brought to British attention by a Booker list that was full of surprises, and also gave a welcome push to small publishers. Ironically, the littlest fish on the shortlist, Graeme Macrae Burnet’s His Bloody Project, published by the crime wing of Scottish indie Saraband, turned out to be the most accessible and commercially successful of the set: it’s a puzzle about murder and motivation in a 19th-century crofting community that keeps the reader eagerly guessing.

Paul Beatty’s Booker-winner The Sellout had clocked up 18 rejections from cautious publishers

Some bold experimenters returned this year. Eimear McBride followed A Girl Is a Half-formed Thing with a coming-of-age story about an Irish drama student in London, The Lesser Bohemians (Faber). Though some found the contrast between her fractured main narrative voice and the inclusion of a more conventionally told story of abuse jarring, with only her second novel she has achieved the near-impossible – finding a new way to write about sex and intimacy. Paul Kingsnorth continued his Buckmaster trilogy in Beast (Faber), jumping 1,000 years from the Norman England of The Wake to a man alone on a moor, trying to vanquish his own insecurities. Rachel Cusk’s Transit (Jonathan Cape), the second instalment in a trio of novels that channel disparate voices through the consciousness of their narrator, was even better than 2014’s Outline.

Facebook Twitter Pinterest First novel in a decade … Jonathan Safran Foer. Photograph: Jeff Mermelstein

China Miéville moved from SF towards fable with the eerie, ungraspable This Census-Taker (Picador), while cult Irish writer Mike McCormack made a welcome return in Solar Bones (Tramp Press), the swirling, single-sentence reminiscence of an ordinary man on his last day of life – a deserving winner of the Goldsmiths prize for innovative fiction. On the Booker shortlist, David Szalay pushed at the fault lines between the novel and short story form in All That Man Is (Jonathan Cape): linked tales of European masculinity in crisis, whose effect is monumentally bleak, but which contain some of the best prose to be found in English this year. And Deborah Levy’s Hot Milk (Hamish Hamilton) plumbed mythic and psychoanalytical archetypes for a profound exploration of a damaged mother-daughter relationship.

Amid a promising array of unconventional debuts, two stood out for me: Nothing on Earth by poet Conor O’Callaghan (Doubleday Ireland), a very contemporary slice of gothic exploring the atomisation of modern life, set on an Irish ghost estate, and Infinite Ground by Martin MacInnes (Atlantic), a metaphysical mystery about a man’s disappearance with shades of Conrad and Borges. Both show enormous poise and control.

Sarah Perry's The Essex Serpent is the book I’ll be wrapping most copies of this Christmas

Sarah Perry made an unusual journey with her second novel: from her uncanny 2014 debut, After Me Comes the Flood, to something much more outwardly conventional, but sacrificing none of her singular sensibility. Set in the 1890s, The Essex Serpent (Serpent’s Tail) – a gloriously rich tale of folklore and science, myth and belief, friendship and love – is the book I’ll be wrapping most copies of this Christmas, and comes with a gorgeous cover to boot.

There were high-concept successes from Colson Whitehead, who made the network for escaping slaves a real system of train tunnels in The Underground Railroad (Fleet), a charged and important novel that pushed at the boundaries of fiction at the same time as documenting historical atrocity, and Naomi Alderman, who imagined women as the stronger sex in The Power (Viking). The result is as clever and thought-provoking as it is furiously readable.

Facebook Twitter Pinterest Gloriously rich tale of friendship, love and folklore … Sarah Perry. Photograph: Graeme Robertson/The Guardian

Álvaro Enrigue’s endlessly inventive Sudden Death (Harvill Secker, translated by Natasha Wimmer) explored the bloody clashes of Spanish empire v Aztec, old world v new, through the medium of a tennis match played by Caravaggio. At the other end of the literary spectrum, Jo Baker’s cool, restrained A Country Road, A Tree (Doubleday) performed the unlikely feat of successfully channelling Samuel Beckett’s prose style for a powerful biographical novel that illuminated his work with the French resistance and the difficult fruition of his genius.

We said goodbye to one master of the short story, William Trevor, who died in November; and the same month saw the introduction to the UK of a leading American practitioner of the form: The Visiting Privilege by Joy Williams (Tuskar Rock) showcases her boundless talent in stories that are darkly funny, ruthless and compassionate by turns.

Finally, if Christmas prompts thoughts of escape, here are two books guaranteed to transport you to other times and places. Francis Spufford’s fiction debut, Golden Hill (Faber), is brilliant, and brilliant fun: a loving recreation of New York in 1746, where a mysterious stranger arrives from England, and gets into ever-deeper trouble. And despite a longstanding suspicion of whimsy and fluffy animals, I was charmed beyond measure by Kevin MacNeil’s The Brilliant & Forever (Polygon), a surreal comedy centring on a literary festival on a Scottish island where humans and talking alpacas uneasily coexist. In difficult times, a story-writing alpaca never goes amiss.

• Save at least 30% on this year’s critics’ choices when you buy at the Guardian Bookshop. Visit bookshop.theguardian.com or call 0330 333 6846. Support the Guardian and its journalism with every book you buy this Christmas. *Free UK p&p for online orders over £10. Minimum £1.99 p&p applies to telephone orders.

Best book lists of 2016