Signs and wonders: Paul Nash's Landscape of the Megaliths, featured in Adam Thorpe's On Silbury Hill. Image: Lauren McLean/V&A Images

We recognise that history is written by the winners but are slower to see how they command geography. The land reshapes under a coloniser’s eye and the map makes the new vision official. In The Cartographer Tries to Map a Way to Zion (Carcanet, £9.95), the Jamaican writer Kei Miller shows us that the map is not the territory. His sparkling, accessible and moving poems won this year’s Forward Prize for Best Collection. Olivia Laing’s The Trip to Echo Spring: on Writers and Drinking (Canongate, £10.99) maps the psyche of the great literary drunks of America: Fitzgerald, Hemingway and their merry peers. Grimly educational, gorgeously written.

Michael Schmidt’s The Novel: a Biography (Harvard University Press, £29.95) is a wonderful, opinionated and encyclopaedic book that threatens to drive you to a lifetime of rereading books you thought you knew and discovering books you know you don’t. And for a very brief but enormously timely essay on why we need better ways of thinking about our past, The History Manifesto by Jo Guldi and David Armitage (Cambridge University Press, £12.99) is excellent. It demonstrates why we need history – in universities but also in public discourse – and wonders provocatively why governments second economists and the like to their service but never historians, who have a distinct responsibility to underline the complexity of social causation, the need for long views and the dangers of narrating the past from inside one or another kind of silo.

Roy Jenkins wasn’t the man he seemed to be. His father had indeed been a coal miner but by the time his only child was born he’d moved far from the pit village. As a full-time trade union official he lived a middle-class lifestyle, complete with a live-in maid. Praised for his war service, the closest he came to conflict was when he managed to kill the colonel’s driver accidentally during gunnery training. And the bon vivant womaniser who legalised homosexuality had, it seems, been involved in a passionate love affair with Tony Crosland at Oxford. John Campbell’s Roy Jenkins: a Well-Rounded Life (Jonathan Cape, £30) is a page-turner of a biography, written with subtlety, clarity and considerable skill.

Winding road: a tram makes its way through the streets of Kolkata in the 1960s, the setting for Neel Mukherjee's The Lives of Others

Two splendid novels stand out for me this year, both of which I have read more than once and shall read again. They are Neel Mukherjee’s The Lives of Others (Chatto & Windus, £16.99) and Ali Smith’s How to Be Both (Hamish Hamilton, £16.99). The Lives of Others is set in and around Kolkata in the 1960s – it is a powerful and intricate study of personal and political life. How to Be Both is innovative in structure – two stories, one about a Renaissance artist and one about a modern girl in mourning, which can be read in either order. They illuminate each other in constantly unexpected ways. The novel is alive and kicking – and changing.

Sheri Fink’s Five Days at Memorial (Atlantic, £9.99) is the gripping and complex account of what happened in a New Orleans hospital in the immediate aftermath of Hurricane Katrina. It has some similarities with J Anthony Lukas’s Common Ground, a classic of American non-fiction first published in 1985. Focusing on three families, two white and one black, Lukas tells the story of busing in Boston in the 1970s. In the process an entire history of race in the US unfolds before our eyes. Amazingly, this masterpiece is not currently in print in the UK.

I had the great honour of taking part in a discussion at this summer’s International Book Festival in Edinburgh with Sir Tom Devine, who has recently republished his excellent trilogy on Scottish history. I’ve taken great pleasure in reading Sir Tom’s work over the years and, although I haven’t had much time in recent months, I always enjoy revisiting The Scottish Nation: a Modern History (Penguin, £12.99) when I can. It covers around 300 years, from the late 17th century to the present day, and teaches us how we can use Scotland’s past to better understand our future. It’s not a secret that I am a historian at heart and I think that this book takes its place among the greatest accounts of Scotland’s history available today.

Damon Galgut’s Arctic Summer (Atlantic, £17.99), a novel about how E M Forster came to write A Passage to India, is an extra­ordinary piece of work – unremittingly beautiful, the period invested with thrilling life and Morgan himself emerging as a creation of immense empathy. Set in the gambling dens of Macau, Lawrence Osborne’s The Ballad of a Small Player (Hogarth, £12.99) is a perfectly written existential thriller, a spooky, gripping, heart-in-your-mouth read that has profound things to say about the only god who rules human affairs – chance. Finally, Jane Gardam’s The Stories (Little, Brown, £20) is a collection spanning her entire career that reminds us, if any reminder were needed, that she is one of the finest living writers in the English language.

In No Good Men Among the Living (Metropolitan, £17.99), Anand Gopal has produced the best piece of investigative journalism to come out of Afghanistan in the past 12 years. He has a deep knowledge of the rural south, took great risks in visiting some of the most dangerous insurgent areas and has conducted thousands of hours of interviews. He shows us the surreal gap between the rhetoric of the “international community” in Afghanistan and the reality on the ground. We meet Taliban commanders, women in remote communities, warlords and US special forces and watch lives intersect in dizzying coincidences spanning Helmand and Guantanamo. In the process, Gopal exposes how shallow our jargon about “development, governance and state-building” has been. His book should be a model for all our analysis of intervention, from Libya to contemporary Iraq.

The Chambers Dictionary recently announced that its word of 2014 is “overshare”. This impetus to put too much personal stuff online powers the narrative in Dave Eggers’s The Circle (Penguin, £7.99), a story that will come back to haunt you every time you log on to social media. This tale of Mae Holland, a new recruit at a tech company (an amalgam of Apple, Google, Amazon, Facebook, Twitter, et al), and how she is sucked into their mindset made me think about the boundary between privacy and secrecy and what constitutes an interior life in the digital age.

David Reynolds’s The Long Shadow (Simon & Schuster, £9.99) is a masterful study of the aftermath of the First World War and of the many complex and contentious ways it has been remembered, represented and refracted. Poems That Make Grown Men Cry (Simon & Schuster, £16.99) is an anthology edited by Anthony and Ben Holden. The father and son combination uses its well-worn contacts books to gather selections from John le Carré, Clive James, Rowan Williams, Melvyn Bragg and 96 other notable men. There are the inevitable picks – Larkin, Hardy, Housman, Edward Thomas, Wilfred Owen – and some surprises. I didn’t weep reading these poems but I was moved by many of them, not least by an exquisitely sad hokku written by Kaga no Chiyo, a Japanese poetess of the 18th century, about the death of her young son: “Dragonfly catcher,/Where today/have you gone?” The words, Boris Akunin writes, “sound beautiful in the Japanese”.

I’m glad I read Ian McEwan’s The Children Act (Jonathan Cape, £16.99) without having to review it. As with much of his fiction, it explores the tension between cool-headed secularism and ardent belief. It is at times preposterous – and yet it has a magical readability and is slender enough to read in one intense, absorbing sitting.

Little Toller Books, which has been reissuing classics of topographical, nature and landscape writing over the past few years, is now commissioning its own original monographs. Among the first are two new classics, in my opinion: The Ash Tree by Oliver Rackham, who is quite simply our best writer today on the history of landscape, woodland and forest, if not our best ever; and On Silbury Hill by Adam Thorpe, a remarkable and moving mix of history, autobiography and genius loci (both £15). These are compact books packed with erudition, enthusiasm and rapt personal engagement. A wonderful series in the making.

It’s difficult to sum up what a beautiful thing, what a strong, gentle and discerning book, what a lit firework, what a piece of elegance and understanding and what a good story Paul Bailey’s The Prince’s Boy is (Bloomsbury, £16.99). I’ll be telling people to read it for the rest of my life. Among the debuts I read and admired this year, Anna Whitwham’s Boxer Handsome (Chatto & Windus, £12.99) had something outstanding and compelling about it that called to mind the prose of the great Nell Dunn and reminded me of the vital, good fighter that the novel form is.

The American playwright Jeff Jackson’s debut novel, Mira Corpora (Friday Project, £8.99), about a young runaway’s series of nightmare-inducing adventures, including drug-related prostitution, is skilfully told in clear yet somehow utterly disorientating prose. On almost the opposite end of the social scale is the Australian novelist Elizabeth Harrower’s In Certain Circles (Text Publishing, £12.99). Her latest rediscovered gem follows a precocious young debutante from having the world at her feet to life beneath the feet of her domineering husband. Harrower evokes the waste and futility of a decadent class with all the bite and poignancy of F Scott Fitzgerald.

Henry Marsh’s memoir of his life as a neurosurgeon, Do No Harm (Phoenix, £8.99), is a strikingly honest and humane account of what it means to hold the power of life and death in your hands. As he approaches retirement, Marsh is humbled by that power, honest about his failures and valiant in his defence of the NHS. The book is eloquent, edifying and necessary. Paul Kingsnorth’s The Wake (Unbound, £16.99) didn’t quite make it on to our Man Booker shortlist this year but it still stirs and moves me and I remain in awe of its unbounded – pun intended – ambition.

It is becoming a cliché to say, “Paul Ewen is a comic genius and Francis Plug: How to Be a Public Author (Galley Beggar Press, £11) is the funniest book in years” – but it’s no less true. The set-up is simple: in each chapter, the simpleton know-all Francis Plug attends a literary event featuring a former Booker Prize winner, tries to fit in, fails, gets drunk, listens and observes, misbehaves, queues up, gets the former prize­winner to sign his first edition of their Booker novel (all title pages are reproduced), engages in painful dialogue about whatever red herring is swimming through his brain, returns enlightened and perplexed to his other life of gardening and grouching and waiting for literary super­stardom. It might come across as a gentle satire on book folk but Vanity with a capital V also gets an entirely hilarious kicking.

I loved Quiet Dell (Jonathan Cape, £18.99) by Jayne Anne Phillips, a novel based on the true story of a murderer in 1930s West Virginia. The main characters are real, others invented; letters and documents are quoted, conversations imagined. It’s both vividly real and curiously fantastical – improbable even. There’s an investigation, a trial and a love story and, at the heart of it, a woman journalist trying to report on the crime while also doing justice to the victims. It has something of In Cold Blood about it but is more involving, less bleak.

My favourite book of 2014 is Do No Harm, Henry Marsh’s painfully honest memoirs of his work as a brain surgeon, but I might be suspected of bias as I am married to the author. The book that lit up my own brain this year is Thinking Big: How the Evolution of Social Life Shaped the Human Mind (Thames & Hudson, £18.95) by the archaeologists Clive Gamble and John Gowlett and the evolutionary psychologist Robin Dunbar – an interdisciplinary tour de force, testing the “social brain hypothesis” against the archaeological and primatological evidence. It is a triumph of collaboration, as well as a gripping detective story.

Already established as the best American author-reporter since Tom Wolfe, Michael Lewis continues his series of explorations of the financial crash in Flashboys (Allen Lane, £20), which, with a clarity that pops eyes and drops jaws, explains how a group of traders used technology to get vital milliseconds ahead of the market. In a year when revivals of Tennessee Williams’s A Streetcar Named Desire and Cat on a Hot Tin Roof confirmed his greatness among American playwrights, John Lahr’s magnificent biography Mad Pilgrimage of the Flesh (Bloomsbury, £30), though a little skimpy on the significance of the dramatist’s Catholicism, illuminates the family, sexual and creative psychoses that fuelled his rise and decline. Rachel Cusk’s wildly original Outline (Faber & Faber, £16.99) is officially a novel, although it continues her skirmishes on the borders of fictional and autobiographical forms.

The new translation of Dostoevsky’s Crime and Punishment (Penguin Classics, £8.99) will give a jolt to the nervous system to anyone interested in the enigmatic Russian author. This vivid, stylish and rich rendition by Oliver Ready compels the attention of the reader in a way that none of the others I’ve read comes close to matching. Using a clear and forceful mid-20th-century English idiom, Ready gives us an entirely new kind of access to Dostoevsky’s singular, self-reflexive and at times unnervingly comic text. This is the Russian writer’s story of moral revolt, guilt and possible regeneration turned into a new work of art.

Adam Nicolson’s The Mighty Dead (William Collins, £25), a thrilling and complex book, enlarges our view of Homer, who, Nicolson writes, tells us how we have become what we are. He justifies this claim through his own adventures of body and mind, as well as deep research. Odysseus’s near-drowning by the sea god Poseidon is brought alive by Nicolson’s sailing experience. He likens the Greeks at Troy to modern urban street gangs who find the law no good and see revenge as a form of justice. No Greek needed – and there’s something that hits the mark on every page.

John Lanchester describes economics as a conversation rather than a science in How to Speak Money (Faber & Faber, £17.99). He kept me laughing and learning about language and money with his sharp, witty and illuminating remarks. Technical writing as good as this is so rare it deserves applause.

Nothing is more fun than disagreeing noisily with the Man Booker judges but this year they chose the best book, without a doubt. Richard Flanagan’s The Narrow Road to the Deep North (Chatto & Windus, £16.99) confirms his place as one of our finest writers. Meanwhile, Daniel Kehlmann’s profoundly satisfying and disturbing F: a Novel (translated by Carol Brown Janeway, Quercus, £16.99) reveals yet another dimension to this extraordinary writer, best known here for his historical study of the relationship between Carl Gauss and Alexander von Humboldt, Measuring the World. Quite simply, F is a masterpiece that nobody should miss.

In 2014, I discovered the novels of Jean Echenoz – who, though published in France by Les Éditions de Minuit, that bastion of the nouveau roman, is a writer of great narrative accessibility. His masterpiece, Ravel (€13), is an imaginative recreation of the life of the great French composer. But Echenoz turns his tale into a spellbinding theatre of ideas about the lonely solipsism of fame and the ongoing doubt that haunts most creative careers. As someone who is always dazzled by John Eliot Gardiner’s ability to find the revolutionary and revelatory in just about everything he conducts, I was thoroughly gripped by his investigation into the great spiritual and humanistic cathedral that is the work of J S Bach, Music in the Castle of Heaven (Penguin, £10.99).

I was bowled over by Saeed Jones’s Prelude to Bruise (Coffee House Press, £16), a beautiful and biting collection of poetry that has been making waves in the US. Investigating race, sexuality and what it means to be southern, Jones’s lean, searing lines transcend identity politics. I was also captivated by Maggie Nelson’s The Art of Cruelty (W W Norton, £12.99). Nelson is one of the best cultural critics working today and her investigation goes to some dark places, taking in Abu Ghraib and torture porn as well as the work of Bacon, Plath, Artaud and de Sade. Rewarding and thought-provoking.

I like to be taken for a ride, so John Waters’s Carsick (Corsair, £16.99) fits the bill, as America’s most arch ironist hitch-hikes his way across the US. It sounds like a Lou Reed song and so it is, par excellence. I also enjoyed the seasick trip with Sea Shepherd’s anti-whaling vegans in Sam Vincent’s revealing, sometimes hilarious Blood & Guts: Dispatches from the Whale Wars (Black Inc, £19.99). It confirms Paul Watson, their leader, as the anti-Ahab of our time. A yet more visceral, interior journey is Helen Macdonald’s transcendent H Is for Hawk (Jonathan Cape, £14.99), the deserving winner of the Samuel Johnson Prize, mixing deep grief with the natural sublime. Never has the eye of a raptor assumed such fearful, beautiful meaning.

Finally, a voyage into Viktor Wynd’s Cabinet of Wonders (Prestel, £29.99) presents a dreamy, Gothic melange of curiosities and taxidermy – À rebours meets World of Interiors. I like Wynd’s claim that clutter is creative. Surveying the tottering piles of books, bones, birds’ heads and holey stones strewn about my own house, I find this notion most reassuring.

In 2014, as Scots we made a historic choice to stay a part of the United Kingdom. Few of us involved will forget the surging emotions and feelings of the campaign. David Torrance’s book The Battle of Britain (Biteback, £14.99) thankfully adds evidence and facts to our understanding of the referendum decision. Important and authoritative, this book will help future generations explain this generation’s choice.

Not long after the publication of Kit Wright’s new collection of poems, Ode to Didcot Power Station (Bloodaxe, £9.95), part of the station burned to the ground. Wright may sometimes be described as a poet of “light” subjects but he isn’t to be taken lightly. The title poem, a mock-McGonagall paean to the “All-belching amphorae” that “form eternal acid rain”, subverts its own subversion: “DIDCOT, thou bugger!/Thou teaser of the mind/And recollection tugger!” The whole collection is Wright in high flight, full of puns, bullseye bombast and outrageous but never gratuitous rhymes (“dancing or prosaic” with “Deoxyribonucleic” in “The Walk of a Friend”). Everywhere the quotidian – weeds, a statue in Lisbon, an Anglo-Saxon queen of Essex called Ricole – is transformed by his wicked mastery of form.

Online trolling has cropped up in the news this year with the frequency of baby pictures from your new-parent friend on Face­book. But there has been little attempt to grapple with the problem beyond simplistic headlines and polarised views. Should we lock up everyone who has ever been rude online? Should we allow anonymous tweeters to threaten women with rape in the name of free speech? The Dark Net by Jamie Bartlett (William Heinemann, £20) takes a judgement-free look at the mechanisms of trolling and other internet bad behaviour and generates more light than heat.

Distant Neighbors: the selected letters of Wendell Berry and Gary Snyder (Counterpoint Press, $30) showcases these two great American poet-philosophers at most thoughtful and least guarded. Decades of correspondence drill down through the layers of their concerns – land use, economics, farming, wild nature, the life of the spirit, modernity itself – as they struggle to live within a culture tearing itself up by its roots. Those roots are exposed in Ben Marcus’s bleak and brilliant short story collection Leaving the Sea (Granta, £16.99). What begins as a series of realist tales about overweight, disappointed American men ends in a fragmented blur of stories from a cold, distant future that seems all too likely to come upon us faster than we think.

In a year when two proposed chairpersons of an inquiry into alleged establishment cover-ups have had to step down for inadvertently forgetting they were too close to the establishment they were investigating, Owen Jones’s The Establishment (Allen Lane, £16.99) is very timely. Fantastic research and personal encounters with people adversely affected by every one of the establishment’s tentacles make this an eye-opening state-of-the-nation book. Jones untangles the contorted logic of Britain’s powerful elites, who think that because they’re British and fundamentally decent, nothing they do can possibly be wrong.

I am a sucker for any book that advertises itself as the latest Scandinavian crime thriller. Iceland’s most recent contribution to the genre is Arnaldur Indriolason’s Strange Shores (Vintage, £7.99). The narrator is an off-duty detective from Reykjavik who returns to his former home in a remote rural community. He tries to unravel the story behind two simultaneous disappearances in a snowstorm decades earlier. The pace is slow; the characters are mostly old and uncommunicative. Nothing much happens beyond the gradual, painful, extraction of the truth. Not a “thriller” but thoroughly absorbing nonetheless.

There aren’t many writers as confident and competent in fiction as in non-fiction but Simon Sebag Montefiore is one of them. His One Night in Winter (Arrow, £7.99) is a murder mystery built around characters in Stalin’s Kremlin and a group of teenage children of the Soviet elite. He captures perfectly the fragility of a society in which the lives of those in Stalin’s entourage are governed by the terrifying paranoia of the boss and his depraved acolyte Beria. A gripping yarn and a good history at the same time.

Out of time: Portrait of a Man with a Ring by Francesco del Cossa, the painter in Ali Smith's new novel

In a sense, I already chose my book of the year as I was on the jury that awarded the Man Booker Prize to Richard Flanagan’s The Narrow Road to the Deep North (Chatto & Windus, £16.99), which is a superb book and thoroughly deserved to win. But I also loved Ali Smith’s How to Be Both (Hamish Hamilton, £16.99), a marvellous exploration of what it means to look, then look again. Spiralling and twisting stories suggest the ways in which we can transcend walls and barriers – not only between people but between emotions, art forms and historical periods. It is a jeu d’esprit about a girl coming of age and coming to terms with her mother’s death, a ghosting of a Renaissance fresco painter in a 21st-century frame and an exhortation to do the twist.

Dying is a painful, inevitable business. Wonderfully, for a medical man, in Being Mortal: Illness, Medicine and What Matters in the End (Profile, £15.99) Atul Gawande explains with remorseless honesty that our doctors and specialists and nursing homes make it much worse for all of us by drawing out the experience – and he has the examples to prove it. Gawande makes you long for society to move to the point at which, like the Romans, we will be free to choose the time and manner of our going. This isn’t a nice, comforting book but for those of us of a certain age it’s essential reading.

In The Strangest Family (William Collins, £25), Janice Hadlow, a witty storyteller and a thoughtful historian, argues that the British monarchy owes its longevity to an inspired conceptual shift in which the royal family became identified as a model of domestic virtue, a shift originating – this is the funny bit – within the dysfunctional family of George III. Ippolito Nievo’s Confessions of an Italian (Penguin Classics, £12.99), first published in 1867 and now newly translated by Frederika Randall, is a sprawling, exuberantly picaresque novel in which Napoleon has a walk-on part, his face covered in shaving soap, and in which Europe’s revolutions are the backdrop to the adventures of a winningly self-deprecating hero.

Good Cities, Better Lives (Routledge, £34.99) is an inspirational tour de force by the veteran urban planner Peter Hall, who sadly died within weeks of its publication. As befits a one-time adviser to Michael Heseltine as environment secretary, he is bold but practical. Homing in on the housing crisis, he sets out the case for a new generation of garden cities planned as “city villages”, citing examples of radical urban design from across Europe. The planning of new housing and public transport together is a key lesson – which we have still to learn in Britain.

I offer no excuse for nominating Nicholas Wade’s A Troublesome Inheritance: Genes, Race and Human History (Penguin, £12.99) as the most stimulating non-fiction book of the year, notwithstanding the abuse that greeted its appearance and the unwholesome support it has occasionally received – from, for example, some barely literate American white supremacists who believe (wrongly) that Wade offers them some kind of science-based validation. Wade’s argument, which some see as the various genome projects’ grubby secret, is that race exists in the human species, that races are still evolving and that some races are more suited than others to specific kinds of social behaviours. If ever a book triggered anew the old debate about nature and nurture and human society, or deserved to, this is it. Why do Japanese migrants to the US succeed where Haitians generally do not? This book offers at least the beginnings of an answer.

George Prochnik’s biography of the newly remembered literary darling of the 1930s Stefan Zweig, The Impossible Exile (Granta, £20), is a haunting, tragic work – and since I am now bent on reading every Zweig that Pushkin Press, which has almost cornered the British market for his vast output, can push out to satisfy the new curiosity, Prochnik’s book is the ideal vade mecum.

In its centenary year, the First World War has been strongly debated but David Reynolds’s The Long Shadow (Simon & Schuster, £9.99) uses the war – and not particularly its suffering – to interpret much that has happened to the principal combatants since. He displays a mastery of diplomacy, politics, economics and culture and gives us vivid personality portraits and cracking stories to go alongside the analysis. Anyone wanting to understand why Britain is so hesitant about the EU should read this.

A Hologram for the King (Penguin, £8.99) by Dave Eggers is a compassionate diagnosis of the American dream set largely in Saudi Arabia. It is withering about the caprice of moneyed autocratic power and corporate American fecklessness and has much to say, too, about inter-generational misunderstanding. And it is funny.

Imagine Edna O’Brien’s The Country Girls 50 years on, then cross it with Lars von Trier’s Breaking the Waves and a style inspired by James Joyce in his later years and you might glean some impression of Eimear McBride’s A Girl Is a Half-Formed Thing (Faber & Faber, £8.99). Daunting though this pitch might sound, this novel is an astonishingly powerful debut and – once you get past the first few pages – not “difficult” at all, except for its brutal subject matter.

If you love your science fiction to do its science properly, then The Martian by Andy Weir (Del Rey, £7.99) is for you. The Sunday Times described the book as “Robinson Crusoe on Mars”. It’s gripping and you feel like you’re there with the main character in this most hostile of environments.

The most finely crafted new novel I read this year was Ken Kalfus’s Equilateral (Bloomsbury, £8.99), set in the late 19th century, in which a British astronomer oversees the construction of an enormous triangle in the Egyptian desert as a signal of intelligent life to watching Martians. It’s an austerely cerebral but also darkly funny work of historical science fiction. I also enjoyed an older book by another American “writer’s writer”. James Salter’s 1975 novel Light Years (Penguin, £9.99) is the story of a marriage and its slow unravelling. It is both gossipy and desperately beautiful. Salter is one of those writers who can make fireworks go off in your head with a single sentence.

A confession: the environment is one of the issues I know, in my mind, is a potentially existential problem for human civilisation but one I struggle to engage with on a day-to-day basis. It often seems too complicated, too distant, too abstract. Other issues – such as the housing crisis, the lack of secure jobs, or the realities of slash-and-burn austerity – seem far more real. But Naomi Klein’s This Changes Everything (Simon & Schuster, £10) is a desperately needed wake-up call for people like me, exposing why the threat is real and pressing – but also demonstrating why it offers an incredible opportunity to rebuild our world and create a new generation of sustainable jobs. The book is a warning. It has to be heeded.

On your bike: Alan Johnson's second memoir covers his time working for the Post Office

Robert Harris’s exceptional An Officer and a Spy (Arrow, £7.99) stole my first place in my reading year. He brings an abiding scandal into vivid focus as only he can. We all thought we knew the story but we knew far less than the half of it. Harris captures the anti-Semitism that stalks France even to this day; a French government incapable of enduring loss of face in having jailed the wrong man; Alfred Dreyfus’s suffering in his cell on Devil’s Island, hearing the sea but never seeing it. A brilliant read.

Alan Johnson’s account of his childhood, This Boy (Corgi, £7.99), is less a literary masterpiece than a book that gives us the most extra­ordinary insight into growing up in urban poverty in our time. That he rose to become a Labour cabinet minister and possible leader is rendered all the more remarkable by our daily awareness of the overbearing rigidity of Britain’s class system.

The experience of reading Sarah Waters’s The Paying Guests (Virago, £20) lingers in my memory as one of the most sumptuous periods of this summer. Skilfully evoking south London in the 1920s – the story takes place not far from where I live now – the novel segues gracefully from a sizzling romance between two women to nail-biting crime fiction. (Stories about gay relationships are so much more exciting set when they were taboo.) The writing is impeccable. A joy in every respect.

Peter Oborne is not only our most consistently interesting and surprising right-wing political commentator but also a notable cricket historian. Wounded Tiger (Simon & Schuster, £25) is a compelling, superbly researched history of cricket in Pakistan – a territory into which few others have ventured. Too many cricket books are ephemeral but this is one destined to last. As for the other addictive 11-a-side sport, I much enjoyed David Goldblatt’s The Game of Our Lives (Viking, £20), an intensely readable socioeconomic study of English football in the age of globalisation that at its best reminded me of Arthur Hopcraft’s The Football Man, still the nonpareil.

The older you get, the harder it is to recapture the intoxicating sense of discovery that comes when you first read George Eliot, Nabokov, Tolstoy or Colette. But this year it came again when I read Elena Ferrante’s remarkable Neapolitan novels (Europa Editions, £11.99 each), chronicling a friendship between two women, Lila and Elena, that lasts for more than six decades. Fishing the River of Time by Tony Taylor (Text Publishing, £12.99) is an odd, haunting debut memoir by an octogenarian author about fishing, wilderness and grandparenthood.

Roger Scruton’s new book, How to Be a Conservative (Bloomsbury, £20), begins with his father’s socialism and love of England. The two were inseparable but English liberty led Scruton in the opposite direction. He describes his conservatism as a love of home. By home, he means the common inheritance that belongs to “we”, the people, and which grows out of everyday life. He recognises that the market has a corrosive effect on human settlement. Global capitalism is a “kind of brigandage in which costs are transferred to future generations for the sake of rewards here and now” and society should place constraints on the market. Scruton is describing the conservative instinct of socialism. In politics, I part company with him but the destructive impact of liberal economics over the past 30 years requires that we recognise the enduring presence and value of the conservative instinct in society for continuity and familiarity. It will allow us better to understand the importance to people of home, a sense of belonging and a love of country.

Books of the year tend to be submitted too early to acknowledge November and December releases, so it’s only right to single out a book from late 2013, Nina Stibbe’s hilarious Love, Nina: Despatches from Family Life (Penguin, £8.99). The freshest piece of new fiction I read was the 250-page narrative about a gay bookshop that runs through Philip Hensher’s patchwork novel The Emperor Waltz (Fourth Estate, £18.99). A genuine surprise omission from the recent shortlists, it’s Hensher’s third book on the trot – after King of the Badgers and Scenes from Early Life – that hasn’t had its due. I’m eternally grateful to Adam Begley for his diligent and stylish Updike (Harper, £25), which answered a thousand questions.

I like to read non-political books when I can but two books I’ve particularly enjoyed this year are both about US politics. Double Down (W H Allen, £9.99) by Mark Halperin and John Heilemann is a comprehensive account of the 2012 presidential campaign and how it was won. It also tells the story of the Republican campaign, the bizarre antics of some of the party’s primary candidates and how its nominee’s campaign imploded. Hillary Clinton’s memoir, Hard Choices (Simon & Schuster, £20), provides a fascinating insight into the realities of power and diplomacy during her time as secretary of state. It vividly illustrates that foreign policy is far from binary, no matter how hard commentators seek to graft black-and-white solutions on to intractable problems.

Classic, realistic short stories aren’t my cup of tea, neither as a writer nor as a reader. I actually hate the form, the way a single episode is meant to represent so much more than itself – when you have seen it a hundred thousand times, it feels like a trick, no matter how different the episodes are. But this autumn, I read The Emerald Light in the Air by Donald Antrim (Granta, £12.99), a collection of brilliant short stories, some of them very funny but also always tragic, in a heartbreaking way. One of the stories I still think about every day. I haven’t read anything as good as that in years.

Paper Aeroplane: Selected Poems 1989-2014 by Simon Armitage (Faber & Faber, £14.99) reminds us how securely he has established himself as the poet of his generation. It also shows his remarkable range, especially in his stunning translations of Sir Gawain and the Green Knight and the Odyssey.

Ashes in the Wind by Christopher Bland (Head of Zeus, £14.99) is an epic novel rooted in the stories of two Irish families over the past century and taking us into wars, peace and personal triumphs and tragedies with equal authority. It is a vivid testament to the power that the traditional form of the novel still has. It is extraordinarily gripping and moving.

In the flurry of the First World War centenary commemorations, we should not forget the events of 1814-15, when the curtain was finally brought down on two decades of devastating total war in Europe and Napoleon Bonaparte was forced to surrender on French soil. Not only was the cast of leading characters more colourful, they also managed to craft a more durable (if imperfect) peace at the Congress of Vienna. Andrew Roberts’s Napoleon the Great (Allen Lane, £30) is a brilliant example of “great man” history, brimming with personality and the high-octane Bonapartist spirit. Also superb is Michael Broers’s Napoleon: Soldier of Destiny (Faber & Faber, £30), which takes the story up to 1805 and spends more time on the construction of Napoleon’s (somewhat conflicted) personal identity.

I read three extraordinary books this year: Michael Oakeshott’s Notebooks, 1922-86 (Imprint Academic, £50), a treasury of aphorisms and insights gathered by the philosopher over 60 years; Matthew Olshan’s Marshlands (Farrar, Straus and Giroux, $23), a first novel of human suffering and love in a war-torn wilderness, constructed with great verve and brilliance; and a recovered treasure from 1965, John Williams’s justly celebrated Stoner (Vintage, £8.99), whose pathos and power derive from microscopic observation of human emotion and character. Each work has a voice unlike any other.

Late in the day, I know, I have discovered Karl Ove Knausgaard’s My Struggle sequence, the third of which, Boyhood Island (Vintage, £8.99), came out in English this year. I was put off by the title, by the arrogant-looking author photograph and by being told that this was “Proust updated” (the original still works fine). But the only question that counts is: is it really, really good? And it is. This was also the year when I discovered, thanks to kind friends, the wonderful The Radetzky March by Joseph Roth (Granta, £9.99), the story of the dying days of a proud empire. Hmm. I wonder why that felt so very 2014?

Ian McEwan’s The Children Act (Jonathan Cape, £16.99) was absorbing, sure-footed and moving. So how come it wasn’t shortlisted for the Man Booker Prize? Randall Jarrell said that in the golden age, everyone complains about how yellow everything is.

Ellen McBreen’s Matisse’s Sculpture: the Pin-Up and the Primitive (Yale University Press, £35) locates the sources of Matisse’s sculptures, mainly in photographs. For once, you think the art historian is not making things up, flourishing a file index of abstruse scholarship, but giving you the goods – not showing off but showing you where Matisse got his ideas from.

My novel of the year is Richard Flanagan’s The Narrow Road to the Deep North (Chatto & Windus, £16.99). Deservedly the Man Booker winner, it haunts the mind months after reading. The visceral horror of life and death for Aussie soldiers in a prisoner of war camp is balanced with an understanding of the mindset of their Japanese captors that enlarges the reader’s capacity for compassion. My non-fiction book of the year is Andrew Roberts’s Napoleon the Great (Allen Lane, £30): superb narrative history grounded in new research.

Two books that deserve to be on any beautiful coffee table. The Bloomsbury Cookbook (Thames & Hudson, £24.95) by Jans Ondaatje Rolls reveals the life of the Woolfs, the Bells and others through accounts of their meals together. They loved their food, it seems, just as they loved conversation. An exquisite book.

Another beauty is Edinburgh: Mapping the City (Birlinn, £30) by Christopher Fleet and Daniel MacCannell. This lovely volume discusses maps from the earliest days of Edinburgh to more recent times. It is a fascinating story and a triumph of the book designer’s art – one to delight every map enthusiast.

I have read three first-rate new history books this year, all of them published by Allen Lane, which seems to be cornering the market in such products. In the centenary year of the outbreak of the Great War, two of them were on related subjects. Alexander Watson’s Ring of Steel: Germany and Austria-Hungary at War, 1914-18 (£30) tells the story of the conflict from, as it were, the other side and will be revelatory to most British readers.

Adam Tooze, whose earlier writings have shown a sure grasp of the history of economics and economic policy, brings that talent to bear in Deluge: the Great War and the Remaking of Global Order, 1916-31 (£30). Finally, Tristram Hunt’s Ten Cities That Made an Empire (£25) is a scholarly, deeply researched and well-written study of the British empire in its heyday, seen from an original angle.

One of the most interesting books for me this past year has been the latest “novel” by the much-laurelled Norwegian Dag Solstad. Since the book, known familiarly over there as “the Telemark novel” (its full title is long), does not exist in English, I have been struggling, happily, to make what I can of it in Norwegian (it’s published by Forlaget Oktober). A most peculiar form of fiction, consisting as it does almost purely of fact and dense with detailed genealogies of Solstad’s ancestors, it is animated, nevertheless, by strangely compelling incidents of centuries-old conflict and joy and by the author’s impassioned investigative spirit and wry commentary.

Marilynne Robinson’s Lila (Virago, £16.99)is the third of her magisterial trilogy set in the small Iowa town of Gilead. Fans have been waiting on tenterhooks to revisit the haunting prose style and the elusive world-view of its protagonists: this time, the young girl who marries the widower John Ames. Subtle shifts of loyalties, strange moral priorities make her books compellingly powerful. Private Island by James Meek (Verso, £12.99) is a series of essays that documents with forensic skill the ruthless and chaotic privatisation of Britain. This is the definitive account of how so much has gone and continues to go wrong with Britain’s institutions. Don’t read it all at once – it’s too depressing.

Witty, clever and richly informative, Dave Goulson’s A Buzz in the Meadow (Jonathan Cape, £16.99) strikes the perfect balance between personal anecdote and serious science to encourage us to care for the least-loved members of British nature – insects.

Overtaken to some extent by developments in the past 20 years, Marion Shoard’s This Land Is Our Land (Gaia, out of print) is still a patiently constructed and authoritative classic on the injustices associated with landownership and land access in Britain. A must for anyone who cares about nature, land, politics, law and life in this country.

Two very different books have impressed me this year. The first is Larry Siedentop’s masterpiece, Inventing the Individual (Allen Lane, £20). Siedentop’s ambition is breathtaking and his achievement extraordinary. He takes the reader on a 2,000-year-long journey from classical antiquity to the Renaissance and argues that the diminished liberalism of the 21st century will fade away unless it acknowledges its Christian heritage. Alan Johnson’s Please, Mister Postman (Bantam, £16.99), the sequel to his memoir of a poverty-stricken childhood, is witty, self-deprecating, sometimes uproariously funny and sometimes unbearably sad. It shines like a candle in the naughty world of inauthentic politicians and public alienation.

Jonathan Israel’s Revolutionary Ideas (Princeton University Press, £27.95) is a racy account of the concepts that shaped the French Revolution and its people. Israel captures the excitement and gathering political momentum of the various stages of the revolution, from constitutional monarchy supported by the Marquis de Lafayette, through democratic republicanism articulated by Tom Paine and his followers, to the populist authoritarian rule of Robespierre and Marat. He does not hide from the complexity of the political dynamics involving the crown, the Church, elections, insurrections, foreign military interventions and the Terror. The book leaves the reader with a strong impression of the power of ideas that unlock political energy and the strength of leadership needed to withstand fickle popular opinion.

My book of the year is Denis Johnson’s The Laughing Monsters (Farrar, Straus & Giroux, $25; published in the UK by Harvill Secker in February 2015), about an intelligence agent in Sierra Leone. I love everything Johnson does. I never thought I’d love anything more than Angels or Jesus’ Son but I do think I love this one the best.

This year, a tranche of books were published that move the radical argument significantly further into the mainstream and help put pressure on Labour to adopt bolder policies to reduce inequality. Owen Jones’s lucid and highly readable The Establishment (Allen Lane, £16.99) takes the modern socialist case to its widest audience for decades. Mary O’Hara’s Austerity Bites (Policy Press, £19.99) is a forensic account of the true impact of austerity, while Danny Dorling’s Inequality and the 1% (Verso, £12.99) takes an empirical look at how the lives of the richest damage the rest of society. Should you want sheer emotional power, you could read Harry’s Last Stand (Icon, £8.99), in which the 91-year-old Harry Leslie Smith reminds us what a society without good public services actually looks and feels like.

The two outstanding novels I read this year were Donna Tartt’s The Goldfinch (Little, Brown, £8.99) and Martin Amis’s The Zone of Interest (Jonathan Cape, £18.99). Both are stylistic masterpieces, written with a fluency and an assurance that set them apart from works of lesser quality. Both are artfully constructed and keep the reader guessing; both raise complex moral issues. Tartt’s novel centres on deception, fraud, theft and blackmail, Amis’s on the possibility and meaning of love in the midst of almost unimaginable evil.

Hermione Eyre’s Viper Wine (Jonathan Cape, £14.99) is a trippy spin on the historical novel based on the story of Venetia Stanley, a 17th-century beauty who died of “viper wine” (imagined here to be a cocktail of viper’s blood, the “stale” of pregnant mares and opium – and its effects to resemble those of Botox, lip augmentation and breast implants). Eyre’s prose is an equally wild cocktail, in which David Bowie lyrics are stirred into an impersonation of the language of the Caroline court. The horrors of the beauty industry are taken apart with feline wit and the book will make you purr with pleasure.

Colin Jones’s The Smile Revolution in 18th-Century Paris (Oxford University Press, £22.99) shows what a talented historian can do with the most unlikely material. It examines how the changing politics of France’s most turbulent century were reflected in the nation’s teeth: from the imperiousness of Louis XIV’s Versailles to the winning smiles of sensibility heralded by Rousseau, which were in turn wiped from French faces by the revolutionaries. Politics, literature, dentistry and art are wrapped up in a brilliant piece of scholarship.

Rock Stars Stole My Life! (Coronet, £18.99) made me laugh out loud. The confessions of the music writer Mark Ellen, the sometime editor of Smash Hits and TV presenter of the crap late-night bit of Live Aid, it is also an ode to two casualties of technology: Grub Street journalism and the singles chart.

We’ve had DeLillo, McInerney and Updike on 9/11. Bleeding Edge (Vintage, £8.99) is Thomas Pynchon’s playful and paranoid New York noir. Read it in preparation for Inherent Vice, the master’s first outing on the big screen. Comic book fans should not miss The Borgias (Dark Horse, £45), Milo Manara’s and Alejandro Jodorowski’s graphic epic, published for Christmas in a single volume for the first time.

Ali Smith’s thrilling How to Be Both (Hamish Hamilton, £16.99) was the best novel I read this year by a country mile but as that already has plenty of champions, I’ll take the opportunity to doff my cap to Colin Barrett’s Young Skins (Jonathan Cape, £14.99), a debut short story collection that captures what it’s like to be young in rural Ireland in 2014 – less of the turf fires and contemplative pints of Guinness and more of the drunk stand-offs, failed courtships and boy racers tearing up the town.